Oct 30 00:04:11.626962 ip-10-0-138-81 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:04:11.981616 ip-10-0-138-81 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:11.981616 ip-10-0-138-81 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:04:11.981616 ip-10-0-138-81 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:11.981616 ip-10-0-138-81 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:11.985254 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985171 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:11.985254 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985255 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985259 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985263 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985266 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985268 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985271 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985274 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985277 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985279 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985282 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985285 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985288 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985290 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985294 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985297 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985299 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985302 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985305 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985307 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:11.985323 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985310 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985319 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985323 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985326 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985328 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985331 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985334 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985337 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985340 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985342 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985345 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985347 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985350 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985353 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985355 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985358 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985360 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985363 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985365 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985369 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:11.985785 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985372 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985374 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985377 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985379 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985382 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985384 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985388 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985391 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985394 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985396 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985399 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985401 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985404 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985407 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985411 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985414 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985417 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985419 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985422 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:11.986252 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985424 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985427 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985429 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985432 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985435 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985437 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985440 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985443 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985445 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985448 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985451 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985453 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985457 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985462 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985465 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985468 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985472 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985475 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985478 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:11.986751 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985482 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985485 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985487 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985490 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985493 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985496 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985499 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985501 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985504 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985507 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985509 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985512 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985515 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985517 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985520 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985523 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985525 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985528 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:11.987198 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985533 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985536 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985538 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985541 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985544 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985546 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985548 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985551 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.985555 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986950 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986960 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986963 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986966 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986969 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986972 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986976 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986978 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986981 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986984 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:11.987662 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986987 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986990 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986992 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986995 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.986999 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987002 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987004 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987007 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987010 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987012 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987015 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987017 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987020 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987022 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987025 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987027 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987030 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987033 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987036 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:11.988121 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987038 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987041 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987044 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987046 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987049 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987052 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987054 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987057 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987060 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987063 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987066 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987068 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987071 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987074 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987076 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987079 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987082 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987084 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987093 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987096 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:11.988609 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987099 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987101 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987104 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987108 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987111 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987113 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987116 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987118 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987121 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987123 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987126 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987128 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987131 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987134 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987136 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987139 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987141 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987144 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987147 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987149 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:11.989112 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987152 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987154 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987157 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987160 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987162 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987165 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987167 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987171 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987175 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987178 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987181 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987191 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987196 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987198 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987201 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987204 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987207 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987209 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:11.989589 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987212 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987215 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987217 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987220 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987222 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987224 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987227 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987230 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987232 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987236 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987239 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987241 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987244 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987247 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987250 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987253 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987255 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987258 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987781 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:11.990040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987786 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987789 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987792 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987794 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987797 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987806 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987809 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987811 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987815 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987817 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987820 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987823 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987826 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987828 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987831 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987833 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987836 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987839 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987841 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:11.990493 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987844 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987847 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987849 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987852 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987855 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987861 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987863 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987866 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987868 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987871 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987874 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987876 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987879 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987882 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987885 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987888 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987891 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987893 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:11.990952 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987903 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987906 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987909 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987912 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987914 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987917 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987920 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987922 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987925 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987928 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987930 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987933 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987936 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987938 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987941 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987946 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987948 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987951 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987954 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:11.991403 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987956 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987959 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987961 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987964 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987967 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987969 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987971 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987974 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987977 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987979 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987982 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987985 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987987 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987990 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987993 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987995 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.987998 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988001 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988003 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988006 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:11.991873 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988009 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988011 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988013 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988016 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988018 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988021 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988023 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988027 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988031 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988034 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988036 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988039 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988041 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988044 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988047 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988049 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988052 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988055 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988057 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:11.992354 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988060 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988064 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988068 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988071 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988074 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988076 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988079 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988081 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.988086 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988908 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988919 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988927 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988932 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988937 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988940 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988945 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988949 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988953 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988956 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988959 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988963 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988968 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:04:11.992801 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988971 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988974 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988978 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988981 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988984 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988988 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988991 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988995 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.988998 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989002 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989006 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989010 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989013 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989016 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989020 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989023 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989026 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989029 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989032 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989037 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989041 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989044 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989047 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989050 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989053 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:04:11.993354 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989058 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989062 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989065 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989068 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989071 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989075 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989080 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989083 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989086 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989089 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989092 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989096 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989099 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989102 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989105 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989107 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989112 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989115 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989119 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989122 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989125 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989128 2580 flags.go:64] FLAG: --help="false" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989131 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-81.ec2.internal" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989135 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:04:11.994013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989138 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989141 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989144 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989148 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989151 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989154 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989157 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989160 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989163 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989166 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989169 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989172 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989175 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989180 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989183 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989186 2580 flags.go:64] FLAG: --lock-file="" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989189 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989192 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989195 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989201 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989204 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989208 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989210 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 00:04:11.994602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989214 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989217 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989220 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989223 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989228 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989231 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989235 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989238 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989241 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989244 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989247 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989250 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989253 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989257 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989265 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989267 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989271 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989274 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989277 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989279 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989282 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989285 2580 flags.go:64] FLAG: --port="10250" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989290 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989293 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01eb03f0cf5adc396" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989296 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989299 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:04:11.995162 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989302 2580 flags.go:64] FLAG: --register-node="true" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989306 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989310 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989313 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989315 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989318 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989323 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989326 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989329 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989332 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989335 2580 flags.go:64] FLAG: --runonce="false" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989339 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989342 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989345 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989348 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989350 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989354 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989357 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989360 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989363 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989368 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989371 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989374 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989377 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989380 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:04:11.995859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989383 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989388 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989391 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989395 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989399 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989402 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989405 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989408 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989411 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989414 2580 flags.go:64] FLAG: --v="2" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989419 2580 flags.go:64] FLAG: --version="false" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989429 2580 flags.go:64] FLAG: --vmodule="" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989433 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989437 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989609 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989614 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989617 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989620 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989623 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989625 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989628 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989631 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989633 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:11.996473 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989636 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989639 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989641 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989644 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989646 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989650 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989653 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989655 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989658 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989660 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989663 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989666 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989670 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989673 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989676 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989679 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989681 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989684 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989687 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:11.997026 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989690 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989692 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989695 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989697 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989700 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989703 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989705 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989708 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989713 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989716 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989718 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989721 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989723 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989726 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989730 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989732 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989735 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989738 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:11.997482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989741 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989743 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989746 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989749 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989751 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989754 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989758 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989760 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989763 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989767 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989771 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989774 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989777 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989779 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989782 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989785 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989787 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989790 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989792 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:11.997954 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989795 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989797 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989800 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989802 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989805 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989808 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989811 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989813 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989816 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989820 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989822 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989825 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989827 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989830 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989833 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989836 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989839 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989842 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989844 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:11.998443 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989848 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989851 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989854 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989856 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989859 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989862 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989864 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989867 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989869 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989872 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989874 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989877 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989880 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989882 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989886 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989890 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989894 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989897 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989899 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989902 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:11.998909 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989904 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.989930 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989966 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989971 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989974 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989977 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989981 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989985 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989988 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989991 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989994 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.989997 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990000 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990003 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990006 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:11.999382 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990008 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990011 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990014 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990017 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990019 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990022 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990025 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990028 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990030 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990033 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990035 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990038 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990040 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990043 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990046 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990049 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990052 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990055 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990057 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:11.999817 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990060 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990064 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990066 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990069 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990071 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990074 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990077 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990080 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990082 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990085 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990089 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990091 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990094 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990097 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990099 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990103 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990107 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990110 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990113 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990116 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:12.000257 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990118 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990121 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990123 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990126 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990129 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990131 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990134 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990136 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990139 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990142 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990144 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990147 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990149 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990153 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990156 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990158 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990161 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990164 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990166 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990169 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:12.000740 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990172 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990175 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990178 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990181 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990183 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990186 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990189 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990192 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990195 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990197 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990199 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990202 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990204 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990207 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990209 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990212 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990215 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990217 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990220 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:12.001202 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990222 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990225 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990228 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990230 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990233 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990235 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990240 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990243 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990245 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990248 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990250 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990253 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990255 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990258 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.990261 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.996983 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.996999 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.997013 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.997018 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997076 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:12.001693 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997082 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997086 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997089 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997092 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997095 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997098 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997101 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997104 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997107 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997110 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997113 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997116 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997119 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997121 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997124 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997126 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997129 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997131 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997134 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997136 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:12.002156 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997139 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997142 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997144 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997147 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997149 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997153 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997156 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997160 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997165 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997168 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997171 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997174 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997177 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997180 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997182 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997185 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997188 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997190 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997193 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:12.002655 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997196 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997198 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997201 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997203 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997206 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997209 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997211 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997214 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997217 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997219 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997222 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997224 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997227 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997229 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997232 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997234 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997237 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997239 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997242 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997245 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:12.003097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997248 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997251 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997254 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997257 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997259 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997262 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997264 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997267 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997269 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997272 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997275 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997277 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997280 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997283 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997285 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997288 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997290 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997293 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997295 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997298 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:12.003658 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997300 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997303 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997306 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997308 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997311 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997313 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997316 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997319 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997321 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997324 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997327 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997329 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997332 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997335 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997337 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997340 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997343 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997346 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997348 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:12.004379 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997351 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997354 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997357 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997361 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997364 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997367 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997370 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.997397 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997538 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997543 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997546 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997549 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997552 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997554 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997557 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:12.005009 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997561 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997563 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997566 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997568 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997589 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997592 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997594 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997597 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997600 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997603 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997606 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997608 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997612 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997615 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997618 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997622 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997624 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997627 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997630 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997632 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:12.005550 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997635 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997638 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997640 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997643 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997645 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997648 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997651 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997654 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997656 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997659 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997662 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997664 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997667 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997669 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997672 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997675 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997677 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997680 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997683 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:12.006097 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997685 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997688 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997691 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997693 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997696 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997699 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997702 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997704 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997707 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997710 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997712 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997715 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997717 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997719 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997722 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997725 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997727 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997730 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997732 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:12.006712 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997735 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997738 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997740 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997742 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997745 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997748 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997750 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997753 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997755 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997757 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997760 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997762 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997765 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997767 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997770 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997773 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997775 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997778 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997780 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997783 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:12.007177 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997786 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997789 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997791 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997793 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997796 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997798 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997803 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997806 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997810 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997813 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997816 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997819 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997821 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997824 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997827 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997829 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997832 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997835 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997839 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:12.007735 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997842 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997845 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.997869 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997906 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997910 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997913 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997917 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997919 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997923 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997925 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997928 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997931 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997933 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997936 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997939 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997942 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:12.008183 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997945 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997947 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997950 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997952 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997955 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997958 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997961 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997963 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997966 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997969 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997971 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997974 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997977 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997981 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997983 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997986 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997988 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997991 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997993 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:12.008564 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997996 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.997998 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998001 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998003 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998007 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998010 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998013 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998016 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998018 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998021 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998023 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998026 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998029 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998031 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998034 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998037 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998039 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998042 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998044 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:12.009018 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998047 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998049 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998052 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998054 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998056 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998059 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998061 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998064 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998066 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998069 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998072 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998075 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998077 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998083 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998086 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998089 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998091 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998094 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998097 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998100 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998102 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:12.009530 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998105 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998107 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998110 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998112 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998115 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998118 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998120 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998123 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998125 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998128 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998131 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998133 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998136 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998138 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998141 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998143 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998146 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998148 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998151 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:12.010040 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998154 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998157 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998159 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998162 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998164 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998168 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998171 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998173 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998176 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998178 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998183 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998186 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998188 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998191 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:11.998193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:11.998248 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.000183 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.000923 2580 server.go:1014] "Starting client certificate rotation" Oct 30 00:04:12.010485 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.001015 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:04:12.010944 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.001041 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:04:12.010944 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.003330 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:12.010944 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.009236 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:12.017487 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.017471 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:04:12.020490 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.020473 2580 log.go:25] "Validated CRI v1 image API" Oct 30 00:04:12.021392 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.021366 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:04:12.024812 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.024791 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c95b7c3d-9589-41d6-ba9b-9d3e0a1a4c7b:/dev/nvme0n1p4 dc20bbbe-f42b-4ee0-bcc4-a1a4a28b499e:/dev/nvme0n1p3] Oct 30 00:04:12.024867 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.024812 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:04:12.029810 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.029693 2580 manager.go:217] Machine: {Timestamp:2026-10-30 00:04:12.02846163 +0000 UTC m=+0.296603046 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101019 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2de017dc812d63e9557b64fce9f674 SystemUUID:ec2de017-dc81-2d63-e955-7b64fce9f674 BootID:96b0af6c-3d85-41b8-8a87-a293bf82c5b8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:82:8e:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:82:8e:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:97:78:24:8a:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:04:12.029810 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.029808 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:04:12.029929 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.029896 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:04:12.032430 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.032404 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:04:12.032602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.032433 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:04:12.032652 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.032605 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:04:12.032652 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.032614 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:04:12.032652 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.032643 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:12.033142 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.033132 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:12.033780 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.033770 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:04:12.036408 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.036396 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:04:12.036525 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.036516 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:04:12.037108 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.037098 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:04:12.037139 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.037113 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:04:12.037139 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.037125 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:04:12.037139 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.037134 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:04:12.037256 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.037143 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:04:12.037503 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.037488 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:12.040364 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.040341 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:04:12.045602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.045582 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:12.046384 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046371 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:04:12.046421 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046402 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:04:12.046554 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046544 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:04:12.046594 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046559 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:04:12.046594 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046566 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:04:12.046594 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046583 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:04:12.046594 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046589 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:04:12.046594 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.046596 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:04:12.047060 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047052 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:04:12.047088 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047061 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:04:12.047088 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047068 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:04:12.047088 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047074 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:04:12.047088 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047083 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:04:12.047255 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047093 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:04:12.047721 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047712 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:04:12.047758 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.047721 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:04:12.051614 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.051601 2580 server.go:1263] "Started kubelet" Oct 30 00:04:12.051737 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.051705 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:04:12.051791 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.051689 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:04:12.051791 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.051768 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:04:12.052430 ip-10-0-138-81 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:04:12.053781 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.053764 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:04:12.055058 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.055036 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:04:12.058374 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.058347 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:04:12.059004 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.058986 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:04:12.059371 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.059351 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:04:12.059471 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.059453 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.059878 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.059858 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:04:12.059968 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.059948 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:04:12.059968 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.059969 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:04:12.060092 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060079 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:04:12.060092 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060091 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:04:12.060190 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060170 2580 factory.go:55] Registering systemd factory Oct 30 00:04:12.060190 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060186 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:04:12.060463 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060451 2580 factory.go:153] Registering CRI-O factory Oct 30 00:04:12.060463 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060464 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 00:04:12.060542 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060511 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:04:12.060542 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060524 2580 factory.go:103] Registering Raw factory Oct 30 00:04:12.060542 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.060536 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 00:04:12.061349 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.061334 2580 manager.go:319] Starting recovery of all containers Oct 30 00:04:12.072478 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.072435 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:04:12.073765 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.073751 2580 manager.go:324] Recovery completed Oct 30 00:04:12.075559 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.075319 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 00:04:12.078483 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.078468 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:12.080380 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.080362 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:12.080459 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.080396 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:12.080459 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.080410 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:12.081084 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.081069 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:04:12.081084 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.081081 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:04:12.081084 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.081086 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:12.081258 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.081098 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:04:12.081900 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.081883 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:12.081965 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.081908 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:12.081965 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.081917 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:12.082249 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.082234 2580 policy_none.go:50] "Start" Oct 30 00:04:12.082249 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.082250 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:04:12.082342 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.082260 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:04:12.083119 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.083109 2580 policy_none.go:44] "Start" Oct 30 00:04:12.108049 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.108023 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 00:04:12.108049 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.108040 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-81.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:04:12.108782 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.107943 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-81.ec2.internal.18e325d0089d8171 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-81.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-81.ec2.internal,},FirstTimestamp:2026-10-30 00:04:12.051554673 +0000 UTC m=+0.319696091,LastTimestamp:2026-10-30 00:04:12.051554673 +0000 UTC m=+0.319696091,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-81.ec2.internal,}" Oct 30 00:04:12.110716 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.110702 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:04:12.112297 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.112274 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 00:04:12.112297 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.112289 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 00:04:12.114171 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.114136 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:04:12.118361 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.118347 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 00:04:12.118453 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.118378 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:04:12.118453 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.118387 2580 server.go:87] "Starting device plugin registration server" Oct 30 00:04:12.118693 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.118682 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:04:12.118730 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.118695 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:04:12.118840 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.118827 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:04:12.118931 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.118907 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:04:12.118931 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.118916 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:04:12.119738 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.119718 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:04:12.119818 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.119770 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.122469 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.122452 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h8rmj" Oct 30 00:04:12.138243 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.138220 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h8rmj" Oct 30 00:04:12.143952 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.143854 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-81.ec2.internal.18e325d00a555f3f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-81.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-81.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-81.ec2.internal,},FirstTimestamp:2026-10-30 00:04:12.080381759 +0000 UTC m=+0.348523186,LastTimestamp:2026-10-30 00:04:12.080381759 +0000 UTC m=+0.348523186,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-81.ec2.internal,}" Oct 30 00:04:12.197790 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.144078 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 00:04:12.197790 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.197488 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:04:12.197790 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.197671 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:04:12.197790 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.197695 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:04:12.197790 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.197740 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:04:12.214028 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.214002 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:12.220495 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.220479 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:12.221448 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.221432 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:12.221522 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.221465 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:12.221522 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.221479 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:12.221522 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.221506 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.242464 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.242392 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.242464 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.242429 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-81.ec2.internal\": node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.282180 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.282152 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.299099 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.299071 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal"] Oct 30 00:04:12.299173 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.299139 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:12.300117 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.300104 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:12.300188 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.300129 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:12.300188 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.300139 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:12.301368 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.301355 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:12.301515 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.301502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.301551 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.301530 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:12.302139 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.302124 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:12.302139 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.302131 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:12.302262 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.302151 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:12.302262 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.302175 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:12.302327 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.302153 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:12.302327 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.302301 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:12.303550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.303531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.303662 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.303557 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:12.304420 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.304405 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:12.304483 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.304431 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:12.304483 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.304445 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:12.330841 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.330814 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-81.ec2.internal\" not found" node="ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.335700 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.335679 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-81.ec2.internal\" not found" node="ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.362027 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.361995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e8ee27d044b1e7c2240420e5926cb72-config\") pod \"kube-apiserver-proxy-ip-10-0-138-81.ec2.internal\" (UID: \"0e8ee27d044b1e7c2240420e5926cb72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.362027 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.362028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.362214 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.362047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.382626 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.382600 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.462874 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.462846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.462970 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.462878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.462970 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.462894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e8ee27d044b1e7c2240420e5926cb72-config\") pod \"kube-apiserver-proxy-ip-10-0-138-81.ec2.internal\" (UID: \"0e8ee27d044b1e7c2240420e5926cb72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.462970 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.462954 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.462970 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.462959 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.463102 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.463010 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e8ee27d044b1e7c2240420e5926cb72-config\") pod \"kube-apiserver-proxy-ip-10-0-138-81.ec2.internal\" (UID: \"0e8ee27d044b1e7c2240420e5926cb72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.482962 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.482939 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.583813 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.583786 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.635398 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.635371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.639342 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.639060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:12.684664 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.684625 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.785263 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.785230 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.885769 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:12.885688 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:12.962200 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.962171 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:12.997332 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:12.997306 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:13.000472 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.000454 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:04:13.000587 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.000569 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:04:13.000661 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.000624 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:04:13.000661 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.000637 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:04:13.037334 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.037302 2580 apiserver.go:52] "Watching apiserver" Oct 30 00:04:13.060130 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.060098 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:13.060260 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.060134 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:04:13.115517 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.115490 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:04:13.115875 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.115856 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-q84nm","kube-system/global-pull-secret-syncer-2c7g8","openshift-network-operator/iptables-alerter-swbtw","openshift-dns/node-resolver-8vxtm","kube-system/konnectivity-agent-ghj4h","openshift-multus/multus-additional-cni-plugins-q7q8g","openshift-multus/multus-xc5n4","openshift-multus/network-metrics-daemon-msd6l","openshift-network-diagnostics/network-check-target-q9h2g","openshift-ovn-kubernetes/ovnkube-node-wr7jg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs","openshift-cluster-node-tuning-operator/tuned-j2gmp"] Oct 30 00:04:13.117638 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.117614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.118956 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.118929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.119059 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.118992 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:13.120126 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.120111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.121418 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.121402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.123094 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.123077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.124208 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.124188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.124636 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.124616 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:13.124753 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.124738 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:13.125431 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.125413 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.126721 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.126703 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:13.126809 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.126772 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:13.129664 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.129642 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:13.129762 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.129739 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:13.135330 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.134239 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:04:13.135330 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.134359 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:04:13.139767 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.137545 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:04:13.139767 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.137667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.139767 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.137890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.139767 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.137995 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvd28\"" Oct 30 00:04:13.139767 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.138343 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal"] Oct 30 00:04:13.139767 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.138454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.144455 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.144407 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:59:12 +0000 UTC" deadline="2028-07-11 14:27:49.19615288 +0000 UTC" Oct 30 00:04:13.144528 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.144456 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14894h23m36.051699772s" Oct 30 00:04:13.145686 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.145668 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:13.145797 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.145783 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal"] Oct 30 00:04:13.161260 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.161232 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:04:13.167797 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.167776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.167914 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.167804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvfv\" (UniqueName: \"kubernetes.io/projected/a79d2269-64f4-444e-9d57-c081a6744d81-kube-api-access-xnvfv\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.167914 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.167825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-conf\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.167914 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.167839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-dbus\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.167914 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.167855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-multus\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.168097 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.167926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.168097 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168041 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:13.168097 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysconfig\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.168097 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-kubernetes\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.168097 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxl8s\" (UniqueName: \"kubernetes.io/projected/8ce3b143-5c8f-4170-8f84-095f7668c53d-kube-api-access-cxl8s\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.168304 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc54t\" (UniqueName: \"kubernetes.io/projected/557dfb83-6842-4e18-b8e2-3426d7ba17ed-kube-api-access-kc54t\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.168304 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-agent-certs\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.168304 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-daemon-config\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.168304 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-slash\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.168304 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.168304 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-env-overrides\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.168304 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmkp\" (UniqueName: \"kubernetes.io/projected/679a9efd-b724-4566-91ac-ba5000919e94-kube-api-access-bhmkp\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjgg\" (UniqueName: \"kubernetes.io/projected/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-kube-api-access-dwjgg\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-netns\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-host\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/557dfb83-6842-4e18-b8e2-3426d7ba17ed-serviceca\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-netns\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-node-log\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-tmp-dir\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.168550 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29b9r\" (UniqueName: \"kubernetes.io/projected/a37bd471-b857-4907-b357-aae3315a2721-kube-api-access-29b9r\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-konnectivity-ca\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-k8s-cni-cncf-io\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-kubelet\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-cni-binary-copy\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-bin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-systemd-units\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-var-lib-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-kubelet\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-netd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.168978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-kubelet-config\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.169026 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-multus-certs\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-cnibin\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-modprobe-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-system-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-systemd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-script-lib\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-systemd\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-ovn\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-log-socket\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqf26\" (UniqueName: \"kubernetes.io/projected/d080e6f8-404d-4b58-9f16-c4825e28a3c0-kube-api-access-fqf26\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-tmp\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37bd471-b857-4907-b357-aae3315a2721-host-slash\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-system-cni-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-sys\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-tuned\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.169596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37bd471-b857-4907-b357-aae3315a2721-iptables-alerter-script\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-cnibin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-etc-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/557dfb83-6842-4e18-b8e2-3426d7ba17ed-host\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-socket-dir-parent\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-conf-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttcl\" (UniqueName: \"kubernetes.io/projected/692f7a39-6f14-4a82-9990-60fd90fba46b-kube-api-access-zttcl\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-bin\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-registration-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-sys-fs\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-var-lib-kubelet\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-hostroot\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-hosts-file\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.169997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-run\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.170320 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079ae719-a70f-4d46-923c-aeb73ce07125-ovn-node-metrics-cert\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-device-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-lib-modules\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-config\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nww5d\" (UniqueName: \"kubernetes.io/projected/079ae719-a70f-4d46-923c-aeb73ce07125-kube-api-access-nww5d\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-socket-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.170841 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.170274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-etc-kubernetes\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.231858 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.231833 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:04:13.231975 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.231912 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rwtz7" Oct 30 00:04:13.241819 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.241781 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8ee27d044b1e7c2240420e5926cb72.slice/crio-50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857 WatchSource:0}: Error finding container 50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857: Status 404 returned error can't find the container with id 50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857 Oct 30 00:04:13.242439 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.242133 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f00d42d4a021e56a936cb97199ccd7.slice/crio-6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead WatchSource:0}: Error finding container 6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead: Status 404 returned error can't find the container with id 6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead Oct 30 00:04:13.244642 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.244617 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:04:13.244754 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.244633 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:04:13.244875 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.244865 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:04:13.244971 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.244905 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:13.245034 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.244966 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:04:13.245034 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.244967 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:04:13.245034 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.244994 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:04:13.245185 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245100 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:04:13.245185 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245133 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:04:13.245185 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245152 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:13.245374 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245219 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:13.245374 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245232 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:04:13.245483 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245462 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:04:13.245561 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245485 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:04:13.245561 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245506 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:04:13.245561 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245514 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8lkn\"" Oct 30 00:04:13.245762 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.245570 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fmnxz\"" Oct 30 00:04:13.247738 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.247723 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:04:13.251785 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.251764 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:04:13.251785 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.251769 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:04:13.254359 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.254333 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:04:13.257488 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.257469 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kct9w\"" Oct 30 00:04:13.257488 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.257471 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7fsr\"" Oct 30 00:04:13.257897 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.257883 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:04:13.257956 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.257904 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6hlg\"" Oct 30 00:04:13.257999 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.257986 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:04:13.258196 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.258067 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xzk4\"" Oct 30 00:04:13.258196 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.258099 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:13.258196 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.258159 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:04:13.258196 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.258175 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:04:13.258196 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.258174 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k8n8m\"" Oct 30 00:04:13.258196 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.258187 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:04:13.258446 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.258272 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h92x8\"" Oct 30 00:04:13.270577 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.270718 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-config\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.270718 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nww5d\" (UniqueName: \"kubernetes.io/projected/079ae719-a70f-4d46-923c-aeb73ce07125-kube-api-access-nww5d\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.270718 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-socket-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.270718 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-etc-kubernetes\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.270718 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.270718 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvfv\" (UniqueName: \"kubernetes.io/projected/a79d2269-64f4-444e-9d57-c081a6744d81-kube-api-access-xnvfv\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-conf\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270729 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-dbus\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270777 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-etc-kubernetes\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-socket-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-multus\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270870 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-multus\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270904 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-conf\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270943 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-dbus\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysconfig\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.270978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-kubernetes\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.271003 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxl8s\" (UniqueName: \"kubernetes.io/projected/8ce3b143-5c8f-4170-8f84-095f7668c53d-kube-api-access-cxl8s\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.271010 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.271077 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.77105346 +0000 UTC m=+2.039194886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271063 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysconfig\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271084 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-kubernetes\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc54t\" (UniqueName: \"kubernetes.io/projected/557dfb83-6842-4e18-b8e2-3426d7ba17ed-kube-api-access-kc54t\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-agent-certs\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-config\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-daemon-config\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271335 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-slash\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271424 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-slash\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-env-overrides\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271435 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271465 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.271677 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmkp\" (UniqueName: \"kubernetes.io/projected/679a9efd-b724-4566-91ac-ba5000919e94-kube-api-access-bhmkp\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjgg\" (UniqueName: \"kubernetes.io/projected/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-kube-api-access-dwjgg\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271524 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271526 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271565 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-netns\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-netns\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-host\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/557dfb83-6842-4e18-b8e2-3426d7ba17ed-serviceca\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-netns\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-host\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-node-log\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271762 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-netns\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-tmp-dir\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29b9r\" (UniqueName: \"kubernetes.io/projected/a37bd471-b857-4907-b357-aae3315a2721-kube-api-access-29b9r\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271807 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271812 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-node-log\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-konnectivity-ca\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.272509 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-k8s-cni-cncf-io\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271902 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-k8s-cni-cncf-io\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-kubelet\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-cni-binary-copy\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-bin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-systemd-units\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.271993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-var-lib-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-kubelet\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-netd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-kubelet-config\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-multus-certs\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272096 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-systemd-units\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-cnibin\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-modprobe-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-system-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.273505 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272149 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-kubelet\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-systemd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-script-lib\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272313 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-daemon-config\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-env-overrides\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272381 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-bin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272400 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-kubelet-config\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272448 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-var-lib-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272478 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-modprobe-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272489 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-tmp-dir\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272525 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-cnibin\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272538 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-multus-certs\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272545 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272549 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/557dfb83-6842-4e18-b8e2-3426d7ba17ed-serviceca\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272584 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-systemd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272597 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-system-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-kubelet\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.274405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-systemd\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-ovn\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-systemd\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-log-socket\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.272735 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272747 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-ovn\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqf26\" (UniqueName: \"kubernetes.io/projected/d080e6f8-404d-4b58-9f16-c4825e28a3c0-kube-api-access-fqf26\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-tmp\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.272811 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret podName:97b31541-b41e-482e-bda2-afb18784d5d4 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.772794688 +0000 UTC m=+2.040936116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret") pod "global-pull-secret-syncer-2c7g8" (UID: "97b31541-b41e-482e-bda2-afb18784d5d4") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37bd471-b857-4907-b357-aae3315a2721-host-slash\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-log-socket\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-system-cni-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272877 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-konnectivity-ca\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272890 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-netd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-sys\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272927 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272939 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-sys\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.275351 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272942 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37bd471-b857-4907-b357-aae3315a2721-host-slash\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-tuned\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272981 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-system-cni-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.272996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37bd471-b857-4907-b357-aae3315a2721-iptables-alerter-script\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-cnibin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-etc-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/557dfb83-6842-4e18-b8e2-3426d7ba17ed-host\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-socket-dir-parent\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-conf-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttcl\" (UniqueName: \"kubernetes.io/projected/692f7a39-6f14-4a82-9990-60fd90fba46b-kube-api-access-zttcl\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-bin\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-registration-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-sys-fs\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-var-lib-kubelet\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-hostroot\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-hosts-file\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273443 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-cni-binary-copy\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-run\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273501 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-bin\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-script-lib\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273516 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-run\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273571 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273581 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37bd471-b857-4907-b357-aae3315a2721-iptables-alerter-script\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273581 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-socket-dir-parent\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273619 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-conf-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079ae719-a70f-4d46-923c-aeb73ce07125-ovn-node-metrics-cert\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273637 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-var-lib-kubelet\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-device-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273711 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-hostroot\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-lib-modules\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273763 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-hosts-file\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.276562 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273582 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-registration-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273859 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-lib-modules\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273865 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-sys-fs\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273884 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273910 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-device-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273937 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-etc-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.273996 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-cnibin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.274053 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.274066 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/557dfb83-6842-4e18-b8e2-3426d7ba17ed-host\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.275524 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-tmp\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.275603 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-tuned\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.275968 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-agent-certs\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.277112 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.276018 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079ae719-a70f-4d46-923c-aeb73ce07125-ovn-node-metrics-cert\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.301844 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.301810 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc54t\" (UniqueName: \"kubernetes.io/projected/557dfb83-6842-4e18-b8e2-3426d7ba17ed-kube-api-access-kc54t\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.301950 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.301867 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmkp\" (UniqueName: \"kubernetes.io/projected/679a9efd-b724-4566-91ac-ba5000919e94-kube-api-access-bhmkp\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.304338 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.304274 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29b9r\" (UniqueName: \"kubernetes.io/projected/a37bd471-b857-4907-b357-aae3315a2721-kube-api-access-29b9r\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.304338 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.304303 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:13.304338 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.304336 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:13.304511 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.304352 2580 projected.go:196] Error preparing data for projected volume kube-api-access-872tn for pod openshift-network-diagnostics/network-check-target-q9h2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:13.304511 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.304377 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttcl\" (UniqueName: \"kubernetes.io/projected/692f7a39-6f14-4a82-9990-60fd90fba46b-kube-api-access-zttcl\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.304511 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.304415 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn podName:26b4ad3c-8956-4e4a-a68f-2df514e7dc25 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.804395305 +0000 UTC m=+2.072536736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-872tn" (UniqueName: "kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn") pod "network-check-target-q9h2g" (UID: "26b4ad3c-8956-4e4a-a68f-2df514e7dc25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:13.304511 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.304475 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxl8s\" (UniqueName: \"kubernetes.io/projected/8ce3b143-5c8f-4170-8f84-095f7668c53d-kube-api-access-cxl8s\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.305976 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.305949 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqf26\" (UniqueName: \"kubernetes.io/projected/d080e6f8-404d-4b58-9f16-c4825e28a3c0-kube-api-access-fqf26\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:13.306434 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.306408 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nww5d\" (UniqueName: \"kubernetes.io/projected/079ae719-a70f-4d46-923c-aeb73ce07125-kube-api-access-nww5d\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.306527 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.306448 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjgg\" (UniqueName: \"kubernetes.io/projected/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-kube-api-access-dwjgg\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.306727 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.306707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvfv\" (UniqueName: \"kubernetes.io/projected/a79d2269-64f4-444e-9d57-c081a6744d81-kube-api-access-xnvfv\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.462039 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.461944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:13.470934 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.470902 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557dfb83_6842_4e18_b8e2_3426d7ba17ed.slice/crio-59da8546fed618e7a8bdbb760c4edf13232e4ee575caec822fbbae1811c38ca0 WatchSource:0}: Error finding container 59da8546fed618e7a8bdbb760c4edf13232e4ee575caec822fbbae1811c38ca0: Status 404 returned error can't find the container with id 59da8546fed618e7a8bdbb760c4edf13232e4ee575caec822fbbae1811c38ca0 Oct 30 00:04:13.481736 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.481717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:13.489006 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.488982 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37bd471_b857_4907_b357_aae3315a2721.slice/crio-3d199ecc8c554cb0c1ba0bf52821e25cbbc9d4414730154e3b6041445cab9610 WatchSource:0}: Error finding container 3d199ecc8c554cb0c1ba0bf52821e25cbbc9d4414730154e3b6041445cab9610: Status 404 returned error can't find the container with id 3d199ecc8c554cb0c1ba0bf52821e25cbbc9d4414730154e3b6041445cab9610 Oct 30 00:04:13.491085 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.491068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:13.498918 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.498887 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aad4aae_3822_4a95_aa08_ecc8fcce7bee.slice/crio-42cffa649dd705f1bd9819b5d8bee55e2be2b5356657d7762a529ea006da3ec0 WatchSource:0}: Error finding container 42cffa649dd705f1bd9819b5d8bee55e2be2b5356657d7762a529ea006da3ec0: Status 404 returned error can't find the container with id 42cffa649dd705f1bd9819b5d8bee55e2be2b5356657d7762a529ea006da3ec0 Oct 30 00:04:13.502189 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.502162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:13.509744 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.509722 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3740a0a7_2025_4cfb_aa4f_9bcbfaf68780.slice/crio-02d0693d91f8a2f25f45c9c9a1671c2d61baf7a3e85c5a7319f4bd7e3e76f683 WatchSource:0}: Error finding container 02d0693d91f8a2f25f45c9c9a1671c2d61baf7a3e85c5a7319f4bd7e3e76f683: Status 404 returned error can't find the container with id 02d0693d91f8a2f25f45c9c9a1671c2d61baf7a3e85c5a7319f4bd7e3e76f683 Oct 30 00:04:13.526746 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.526728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:13.533569 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.533545 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79d2269_64f4_444e_9d57_c081a6744d81.slice/crio-57c3db016636eaf05d5c01bc874808cf8e1c94d924bf6955af1127c11b992ca7 WatchSource:0}: Error finding container 57c3db016636eaf05d5c01bc874808cf8e1c94d924bf6955af1127c11b992ca7: Status 404 returned error can't find the container with id 57c3db016636eaf05d5c01bc874808cf8e1c94d924bf6955af1127c11b992ca7 Oct 30 00:04:13.537821 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.537802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xc5n4" Oct 30 00:04:13.545795 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.545773 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692f7a39_6f14_4a82_9990_60fd90fba46b.slice/crio-c7bb1ae049df5fd6639e4de1fea18dd7e1517dd7224cc2ed155ddfe71ab9d424 WatchSource:0}: Error finding container c7bb1ae049df5fd6639e4de1fea18dd7e1517dd7224cc2ed155ddfe71ab9d424: Status 404 returned error can't find the container with id c7bb1ae049df5fd6639e4de1fea18dd7e1517dd7224cc2ed155ddfe71ab9d424 Oct 30 00:04:13.560831 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.560810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:13.566931 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.566909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:13.568022 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.568005 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:13.568482 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.568429 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079ae719_a70f_4d46_923c_aeb73ce07125.slice/crio-066c3492bb0be8b1cbbdb58a243bb56bab09bebe0654145d1598ad774576ab25 WatchSource:0}: Error finding container 066c3492bb0be8b1cbbdb58a243bb56bab09bebe0654145d1598ad774576ab25: Status 404 returned error can't find the container with id 066c3492bb0be8b1cbbdb58a243bb56bab09bebe0654145d1598ad774576ab25 Oct 30 00:04:13.577885 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.577855 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679a9efd_b724_4566_91ac_ba5000919e94.slice/crio-f78ce33c22222453b260ca7cab74c28787ed573b92cefd4f084db0715f02e9e5 WatchSource:0}: Error finding container f78ce33c22222453b260ca7cab74c28787ed573b92cefd4f084db0715f02e9e5: Status 404 returned error can't find the container with id f78ce33c22222453b260ca7cab74c28787ed573b92cefd4f084db0715f02e9e5 Oct 30 00:04:13.578085 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:13.578065 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce3b143_5c8f_4170_8f84_095f7668c53d.slice/crio-9c8755fbe46f1353a7dc9613e4e22d11a58c88cb2e485a5b13aa18fddd679c72 WatchSource:0}: Error finding container 9c8755fbe46f1353a7dc9613e4e22d11a58c88cb2e485a5b13aa18fddd679c72: Status 404 returned error can't find the container with id 9c8755fbe46f1353a7dc9613e4e22d11a58c88cb2e485a5b13aa18fddd679c72 Oct 30 00:04:13.614299 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.614246 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:13.778457 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.778370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:13.778457 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.778447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:13.778714 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.778555 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:13.778714 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.778622 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:14.778591804 +0000 UTC m=+3.046733210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:13.779064 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.779040 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:13.779200 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.779091 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret podName:97b31541-b41e-482e-bda2-afb18784d5d4 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:14.779077726 +0000 UTC m=+3.047219133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret") pod "global-pull-secret-syncer-2c7g8" (UID: "97b31541-b41e-482e-bda2-afb18784d5d4") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:13.879454 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:13.879415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:13.879660 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.879544 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:13.879660 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.879568 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:13.879660 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.879598 2580 projected.go:196] Error preparing data for projected volume kube-api-access-872tn for pod openshift-network-diagnostics/network-check-target-q9h2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:13.879833 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:13.879661 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn podName:26b4ad3c-8956-4e4a-a68f-2df514e7dc25 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:14.879641609 +0000 UTC m=+3.147783036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-872tn" (UniqueName: "kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn") pod "network-check-target-q9h2g" (UID: "26b4ad3c-8956-4e4a-a68f-2df514e7dc25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:14.136765 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.136731 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:14.210479 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.210412 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" event={"ID":"8ce3b143-5c8f-4170-8f84-095f7668c53d","Type":"ContainerStarted","Data":"9c8755fbe46f1353a7dc9613e4e22d11a58c88cb2e485a5b13aa18fddd679c72"} Oct 30 00:04:14.234244 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.234162 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:13 +0000 UTC" deadline="2028-05-25 18:47:22.603261593 +0000 UTC" Oct 30 00:04:14.234244 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.234242 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13770h43m8.369024802s" Oct 30 00:04:14.242140 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.242103 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xc5n4" event={"ID":"692f7a39-6f14-4a82-9990-60fd90fba46b","Type":"ContainerStarted","Data":"c7bb1ae049df5fd6639e4de1fea18dd7e1517dd7224cc2ed155ddfe71ab9d424"} Oct 30 00:04:14.273719 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.271739 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghj4h" event={"ID":"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780","Type":"ContainerStarted","Data":"02d0693d91f8a2f25f45c9c9a1671c2d61baf7a3e85c5a7319f4bd7e3e76f683"} Oct 30 00:04:14.285187 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.285147 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-swbtw" event={"ID":"a37bd471-b857-4907-b357-aae3315a2721","Type":"ContainerStarted","Data":"3d199ecc8c554cb0c1ba0bf52821e25cbbc9d4414730154e3b6041445cab9610"} Oct 30 00:04:14.306321 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.306281 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q84nm" event={"ID":"557dfb83-6842-4e18-b8e2-3426d7ba17ed","Type":"ContainerStarted","Data":"59da8546fed618e7a8bdbb760c4edf13232e4ee575caec822fbbae1811c38ca0"} Oct 30 00:04:14.314202 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.314163 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" event={"ID":"0e8ee27d044b1e7c2240420e5926cb72","Type":"ContainerStarted","Data":"50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857"} Oct 30 00:04:14.327261 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.327216 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" event={"ID":"a2f00d42d4a021e56a936cb97199ccd7","Type":"ContainerStarted","Data":"6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead"} Oct 30 00:04:14.344405 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.344364 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" event={"ID":"679a9efd-b724-4566-91ac-ba5000919e94","Type":"ContainerStarted","Data":"f78ce33c22222453b260ca7cab74c28787ed573b92cefd4f084db0715f02e9e5"} Oct 30 00:04:14.359075 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.359033 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"066c3492bb0be8b1cbbdb58a243bb56bab09bebe0654145d1598ad774576ab25"} Oct 30 00:04:14.377184 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.377147 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerStarted","Data":"57c3db016636eaf05d5c01bc874808cf8e1c94d924bf6955af1127c11b992ca7"} Oct 30 00:04:14.395651 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.395558 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8vxtm" event={"ID":"8aad4aae-3822-4a95-aa08-ecc8fcce7bee","Type":"ContainerStarted","Data":"42cffa649dd705f1bd9819b5d8bee55e2be2b5356657d7762a529ea006da3ec0"} Oct 30 00:04:14.788803 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.788710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:14.788803 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.788784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:14.789041 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.788922 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:14.789041 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.789028 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret podName:97b31541-b41e-482e-bda2-afb18784d5d4 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:16.789012586 +0000 UTC m=+5.057154006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret") pod "global-pull-secret-syncer-2c7g8" (UID: "97b31541-b41e-482e-bda2-afb18784d5d4") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:14.789432 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.789411 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:14.789511 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.789466 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:16.789455351 +0000 UTC m=+5.057596771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:14.825511 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.825447 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:14.890878 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:14.890181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:14.890878 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.890394 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:14.890878 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.890420 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:14.890878 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.890435 2580 projected.go:196] Error preparing data for projected volume kube-api-access-872tn for pod openshift-network-diagnostics/network-check-target-q9h2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:14.890878 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:14.890500 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn podName:26b4ad3c-8956-4e4a-a68f-2df514e7dc25 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:16.890478619 +0000 UTC m=+5.158620043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-872tn" (UniqueName: "kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn") pod "network-check-target-q9h2g" (UID: "26b4ad3c-8956-4e4a-a68f-2df514e7dc25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:15.082644 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:15.082587 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:15.198817 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:15.198778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:15.199313 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:15.198924 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:15.199375 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:15.199353 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:15.199473 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:15.199448 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:15.199537 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:15.199524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:15.199626 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:15.199607 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:15.235512 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:15.235462 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:13 +0000 UTC" deadline="2028-07-29 17:16:50.843758806 +0000 UTC" Oct 30 00:04:15.235512 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:15.235502 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15329h12m35.60826207s" Oct 30 00:04:16.807496 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:16.807457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:16.808125 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:16.807529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:16.808125 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.807688 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:16.808125 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.807761 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret podName:97b31541-b41e-482e-bda2-afb18784d5d4 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:20.807741237 +0000 UTC m=+9.075882936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret") pod "global-pull-secret-syncer-2c7g8" (UID: "97b31541-b41e-482e-bda2-afb18784d5d4") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:16.808306 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.808185 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:16.808306 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.808238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:20.808221811 +0000 UTC m=+9.076363235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:16.908717 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:16.908623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:16.908900 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.908886 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:16.908966 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.908907 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:16.908966 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.908921 2580 projected.go:196] Error preparing data for projected volume kube-api-access-872tn for pod openshift-network-diagnostics/network-check-target-q9h2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:16.909092 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:16.909006 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn podName:26b4ad3c-8956-4e4a-a68f-2df514e7dc25 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:20.908966804 +0000 UTC m=+9.177108215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-872tn" (UniqueName: "kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn") pod "network-check-target-q9h2g" (UID: "26b4ad3c-8956-4e4a-a68f-2df514e7dc25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:17.199073 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.198410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:17.199073 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.198637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:17.199073 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:17.198789 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:17.199073 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:17.198888 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:17.199073 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.198949 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:17.199073 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:17.199038 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:17.424214 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.424177 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" event={"ID":"0e8ee27d044b1e7c2240420e5926cb72","Type":"ContainerStarted","Data":"8b41f27cd456d9cb8a4b94dcd27a60e20d5596142e0b043b47751a044a1b101a"} Oct 30 00:04:17.427645 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.427612 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2f00d42d4a021e56a936cb97199ccd7" containerID="1952a754f4745df156211864bc246a9456fadbc0f5714b216c0819a1be720a0d" exitCode=0 Oct 30 00:04:17.427785 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.427719 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" event={"ID":"a2f00d42d4a021e56a936cb97199ccd7","Type":"ContainerDied","Data":"1952a754f4745df156211864bc246a9456fadbc0f5714b216c0819a1be720a0d"} Oct 30 00:04:17.433983 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.433956 2580 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="9c6a58edbc0426daa1823e84d042c4c8cef780c477bf89a10b74156421fe39f8" exitCode=0 Oct 30 00:04:17.434095 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.434003 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerDied","Data":"9c6a58edbc0426daa1823e84d042c4c8cef780c477bf89a10b74156421fe39f8"} Oct 30 00:04:17.502211 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:17.502153 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" podStartSLOduration=4.502133973 podStartE2EDuration="4.502133973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:17.467511243 +0000 UTC m=+5.735652670" watchObservedRunningTime="2026-10-30 00:04:17.502133973 +0000 UTC m=+5.770275401" Oct 30 00:04:18.446290 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:18.446251 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghj4h" event={"ID":"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780","Type":"ContainerStarted","Data":"3d566b29170a6f9645553de60e2d3b31f547b799511d494ff006814912321777"} Oct 30 00:04:18.458309 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:18.458275 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" event={"ID":"a2f00d42d4a021e56a936cb97199ccd7","Type":"ContainerStarted","Data":"7c3f3760a958f90e4193b106d8651550dbd82bff566c744ff48ce6b52be611d4"} Oct 30 00:04:18.516203 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:18.516139 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" podStartSLOduration=5.516120535 podStartE2EDuration="5.516120535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:18.515847745 +0000 UTC m=+6.783989173" watchObservedRunningTime="2026-10-30 00:04:18.516120535 +0000 UTC m=+6.784261963" Oct 30 00:04:18.516390 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:18.516266 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ghj4h" podStartSLOduration=2.181385695 podStartE2EDuration="6.516260152s" totalImagesPullingTime="4.334874457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:18.481832538 +0000 UTC m=+6.749973968" watchObservedRunningTime="2026-10-30 00:04:18.516260152 +0000 UTC m=+6.784401580" Oct 30 00:04:19.198238 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:19.198209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:19.198359 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:19.198334 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:19.198429 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:19.198401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:19.198482 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:19.198467 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:19.198539 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:19.198529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:19.198643 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:19.198627 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:19.466117 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:19.465637 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q84nm" event={"ID":"557dfb83-6842-4e18-b8e2-3426d7ba17ed","Type":"ContainerStarted","Data":"a331e2e08b8578b12fd90e92eeefefc53a2194c82fefd11a3141af09bc8ee154"} Oct 30 00:04:20.472646 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:20.472541 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" event={"ID":"679a9efd-b724-4566-91ac-ba5000919e94","Type":"ContainerStarted","Data":"103c15c8bd337bc43d159d091583449a3b98e75c62f57bab0d43c5376ed39662"} Oct 30 00:04:20.844276 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:20.843644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:20.844276 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:20.843731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:20.844276 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.843773 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:20.844276 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.843851 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret podName:97b31541-b41e-482e-bda2-afb18784d5d4 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:28.843835339 +0000 UTC m=+17.111976757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret") pod "global-pull-secret-syncer-2c7g8" (UID: "97b31541-b41e-482e-bda2-afb18784d5d4") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:20.844276 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.843861 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:20.844276 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.843913 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:28.84389681 +0000 UTC m=+17.112038217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:20.944440 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:20.944406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:20.944614 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.944562 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:20.944614 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.944592 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:20.944614 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.944601 2580 projected.go:196] Error preparing data for projected volume kube-api-access-872tn for pod openshift-network-diagnostics/network-check-target-q9h2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:20.944789 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:20.944645 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn podName:26b4ad3c-8956-4e4a-a68f-2df514e7dc25 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:28.94463304 +0000 UTC m=+17.212774445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-872tn" (UniqueName: "kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn") pod "network-check-target-q9h2g" (UID: "26b4ad3c-8956-4e4a-a68f-2df514e7dc25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:21.116074 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.116025 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:04:21.122565 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.122448 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:04:21.116043535Z","UUID":"4be217da-1403-4168-9c99-2a8e66566365","Handler":null,"Name":"","Endpoint":""} Oct 30 00:04:21.124485 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.124466 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:04:21.124595 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.124495 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:04:21.203602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.198730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:21.203602 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:21.198851 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:21.203602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.198915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:21.203602 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:21.198995 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:21.203602 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.199034 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:21.203602 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:21.199096 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:21.343984 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.343949 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:21.344973 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.344908 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:21.381532 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.381474 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q84nm" podStartSLOduration=3.667162981 podStartE2EDuration="9.38145361s" totalImagesPullingTime="5.714290629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:19.501026891 +0000 UTC m=+7.769168319" watchObservedRunningTime="2026-10-30 00:04:21.38145361 +0000 UTC m=+9.649595038" Oct 30 00:04:21.478093 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:21.478021 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" event={"ID":"679a9efd-b724-4566-91ac-ba5000919e94","Type":"ContainerStarted","Data":"3bd19f77b308d48198f0802134fd493e83c26bbf75131b7c40f4001ec5660069"} Oct 30 00:04:22.482618 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.482314 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" event={"ID":"679a9efd-b724-4566-91ac-ba5000919e94","Type":"ContainerStarted","Data":"b09b878ff5f2cc44940a4ba75ee5b1b1ff9c502e26179e200268a166d743e29e"} Oct 30 00:04:22.484762 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.484730 2580 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="559fee6eb134d509b60fcd28cb49532b2c5c2273a04b5c04c8332107e7f7a396" exitCode=0 Oct 30 00:04:22.484859 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.484819 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerDied","Data":"559fee6eb134d509b60fcd28cb49532b2c5c2273a04b5c04c8332107e7f7a396"} Oct 30 00:04:22.487115 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.486720 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8vxtm" event={"ID":"8aad4aae-3822-4a95-aa08-ecc8fcce7bee","Type":"ContainerStarted","Data":"710d7d38c33ed86ef49dcf4bc59a50d5b5a0b56bc3036f6eb662860617900ef0"} Oct 30 00:04:22.488936 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.488913 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xc5n4" event={"ID":"692f7a39-6f14-4a82-9990-60fd90fba46b","Type":"ContainerStarted","Data":"f742a2d13ff27b81aee3debc3ec11bc16e4ae3e8a11b57bd746da1a50ebabca1"} Oct 30 00:04:22.521810 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.521758 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" podStartSLOduration=2.242830923 podStartE2EDuration="10.521741761s" totalImagesPullingTime="8.278910838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:22.520986349 +0000 UTC m=+10.789127786" watchObservedRunningTime="2026-10-30 00:04:22.521741761 +0000 UTC m=+10.789883189" Oct 30 00:04:22.606670 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.606611 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8vxtm" podStartSLOduration=2.294220862 podStartE2EDuration="10.606593098s" totalImagesPullingTime="8.312372236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:22.556007422 +0000 UTC m=+10.824148848" watchObservedRunningTime="2026-10-30 00:04:22.606593098 +0000 UTC m=+10.874734534" Oct 30 00:04:22.642927 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:22.642868 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xc5n4" podStartSLOduration=2.104085992 podStartE2EDuration="10.642854442s" totalImagesPullingTime="8.53876845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:22.642691123 +0000 UTC m=+10.910832547" watchObservedRunningTime="2026-10-30 00:04:22.642854442 +0000 UTC m=+10.910995894" Oct 30 00:04:23.198585 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:23.198537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:23.198709 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:23.198680 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:23.199086 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:23.199070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:23.199179 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:23.199162 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:23.199237 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:23.199226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:23.199320 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:23.199305 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:23.495137 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:23.495101 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-swbtw" event={"ID":"a37bd471-b857-4907-b357-aae3315a2721","Type":"ContainerStarted","Data":"f9e2488abd29e3ebb5fb5f26d63252421186e17255a10b18aa02365a258568f3"} Oct 30 00:04:23.497918 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:23.497888 2580 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="7fe5c0f647ba4798b6f250e5b7fe6b7a2479028e05d52325fa7d6575df21f8e9" exitCode=0 Oct 30 00:04:23.498159 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:23.498136 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerDied","Data":"7fe5c0f647ba4798b6f250e5b7fe6b7a2479028e05d52325fa7d6575df21f8e9"} Oct 30 00:04:23.530661 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:23.530603 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-swbtw" podStartSLOduration=2.9256153620000003 podStartE2EDuration="11.530584578s" totalImagesPullingTime="8.604969216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:23.529827673 +0000 UTC m=+11.797969101" watchObservedRunningTime="2026-10-30 00:04:23.530584578 +0000 UTC m=+11.798726000" Oct 30 00:04:24.501413 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:24.501379 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" event={"ID":"8ce3b143-5c8f-4170-8f84-095f7668c53d","Type":"ContainerStarted","Data":"38ba89ff5dbc4e6883e1cdb398ad57bc6f5c7a0a2b4658bc50741cd86271d4ea"} Oct 30 00:04:24.503566 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:24.503540 2580 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="c3b69913970b9056e8cbf34e9b8d7bc8820027b443cafe3bbafc6cc17290ff75" exitCode=0 Oct 30 00:04:24.503688 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:24.503601 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerDied","Data":"c3b69913970b9056e8cbf34e9b8d7bc8820027b443cafe3bbafc6cc17290ff75"} Oct 30 00:04:24.577055 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:24.576960 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" podStartSLOduration=2.085444871 podStartE2EDuration="12.576939704s" totalImagesPullingTime="10.491494833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:24.537908478 +0000 UTC m=+12.806049899" watchObservedRunningTime="2026-10-30 00:04:24.576939704 +0000 UTC m=+12.845081138" Oct 30 00:04:25.198365 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:25.198335 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:25.198556 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:25.198427 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:25.199065 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:25.199043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:25.199295 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:25.199272 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:25.199385 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:25.199345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:25.199446 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:25.199425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:26.511124 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:26.510939 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerStarted","Data":"e0ad7954b88d0794e076893aba62038604cbacd0aa084b10396d7653ac5c9b2b"} Oct 30 00:04:27.198734 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:27.198702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:27.198902 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:27.198702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:27.198902 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:27.198836 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:27.199013 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:27.198919 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:27.199013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:27.198702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:27.199093 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:27.199015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:27.517115 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:27.517035 2580 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="e0ad7954b88d0794e076893aba62038604cbacd0aa084b10396d7653ac5c9b2b" exitCode=0 Oct 30 00:04:27.517115 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:27.517069 2580 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="5a10f6f5c34ddf4b413aa9b89504f04cc63344318da9a563c5a17376ec2aa981" exitCode=0 Oct 30 00:04:27.517632 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:27.517116 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerDied","Data":"e0ad7954b88d0794e076893aba62038604cbacd0aa084b10396d7653ac5c9b2b"} Oct 30 00:04:27.517632 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:27.517165 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerDied","Data":"5a10f6f5c34ddf4b413aa9b89504f04cc63344318da9a563c5a17376ec2aa981"} Oct 30 00:04:28.520629 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:28.520597 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"f9800843e617cc1ef778668ccf9fffe5386bd5899f845c9907c16161200aae72"} Oct 30 00:04:28.520938 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:28.520671 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"f750b38be0154919c456c955f6463f5dd3bb54c0610b842a91e047c1897266dd"} Oct 30 00:04:28.525382 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:28.525346 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" event={"ID":"a79d2269-64f4-444e-9d57-c081a6744d81","Type":"ContainerStarted","Data":"8b6c01aac4a601175659f7217acc8498816c7ee21e08cdb28bf710848f1c4138"} Oct 30 00:04:28.587873 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:28.587796 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q7q8g" podStartSLOduration=6.5612354459999995 podStartE2EDuration="16.587783612s" totalImagesPullingTime="10.026548166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:28.575476896 +0000 UTC m=+16.843618324" watchObservedRunningTime="2026-10-30 00:04:28.587783612 +0000 UTC m=+16.855925038" Oct 30 00:04:28.906400 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:28.906182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:28.906400 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:28.906396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:28.906637 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:28.906317 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:28.906637 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:28.906461 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret podName:97b31541-b41e-482e-bda2-afb18784d5d4 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.906448594 +0000 UTC m=+33.174590018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret") pod "global-pull-secret-syncer-2c7g8" (UID: "97b31541-b41e-482e-bda2-afb18784d5d4") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:28.906637 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:28.906497 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:28.906637 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:28.906553 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.906537599 +0000 UTC m=+33.174679018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:29.006872 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.006834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:29.007017 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:29.006980 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:29.007017 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:29.006998 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:29.007017 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:29.007008 2580 projected.go:196] Error preparing data for projected volume kube-api-access-872tn for pod openshift-network-diagnostics/network-check-target-q9h2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:29.007125 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:29.007055 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn podName:26b4ad3c-8956-4e4a-a68f-2df514e7dc25 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:45.007043033 +0000 UTC m=+33.275184451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-872tn" (UniqueName: "kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn") pod "network-check-target-q9h2g" (UID: "26b4ad3c-8956-4e4a-a68f-2df514e7dc25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:29.198414 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.198343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:29.198597 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.198428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:29.198597 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.198445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:29.198597 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:29.198531 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:29.198710 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:29.198666 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:29.198756 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:29.198744 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:29.529957 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.529879 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:04:29.530423 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.530284 2580 generic.go:356] "Generic (PLEG): container finished" podID="079ae719-a70f-4d46-923c-aeb73ce07125" containerID="f9800843e617cc1ef778668ccf9fffe5386bd5899f845c9907c16161200aae72" exitCode=1 Oct 30 00:04:29.530423 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.530361 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerDied","Data":"f9800843e617cc1ef778668ccf9fffe5386bd5899f845c9907c16161200aae72"} Oct 30 00:04:29.530522 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.530436 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"f552699d76e110ecb6da8b50623cc697c058a451981d7ba946e7696669c827a3"} Oct 30 00:04:29.530522 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.530470 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"01447caa3b008489ee58878c7a4db593ad1ecf4139dfe1b92d5fd94f22071be2"} Oct 30 00:04:29.530522 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.530500 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"fc6ba93bfa4da69e8811dc3dbf70faed18ef618dc4300b461d4f41015fc35351"} Oct 30 00:04:29.530633 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:29.530532 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"bef1e0a53617bbd0f7ce32d20f5da4042da5fa6a539a4c44383c19ce97f819ec"} Oct 30 00:04:31.198922 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:31.198839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:31.198922 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:31.198895 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:31.198922 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:31.198905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:31.199430 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:31.198977 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:31.199430 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:31.199060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:31.199430 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:31.199119 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:31.537432 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:31.537356 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:04:31.537748 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:31.537727 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"46f91f1562ce79bdd1af8d9d8e5a1ca5ab52ef265d08905b28575c8227e098e0"} Oct 30 00:04:33.198153 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.198123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:33.198746 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.198224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:33.198746 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:33.198232 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:33.198746 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.198254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:33.198746 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:33.198338 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:33.198746 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:33.198414 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:33.545289 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.545263 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:04:33.545658 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.545633 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"a9fdda8129419106095328f2dd2c28263178ea7730370cdcf47c05447e88bbc9"} Oct 30 00:04:33.545925 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.545905 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:33.546012 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.546000 2580 scope.go:122] "RemoveContainer" containerID="f9800843e617cc1ef778668ccf9fffe5386bd5899f845c9907c16161200aae72" Oct 30 00:04:33.562528 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:33.562381 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:34.059322 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.059286 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:34.124102 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.124072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:34.279803 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.279775 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:34.550854 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.550781 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:04:34.551168 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.551147 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" event={"ID":"079ae719-a70f-4d46-923c-aeb73ce07125","Type":"ContainerStarted","Data":"d559f98a6eadeb222a5649449d37c6960b641db7bc3e711e386e0f540c62c451"} Oct 30 00:04:34.551352 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.551336 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:34.566198 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.566173 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:34.613679 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.613638 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" podStartSLOduration=7.847484813 podStartE2EDuration="22.613625161s" totalImagesPullingTime="14.766140348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:34.61322505 +0000 UTC m=+22.881366478" watchObservedRunningTime="2026-10-30 00:04:34.613625161 +0000 UTC m=+22.881766585" Oct 30 00:04:34.970341 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.970305 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2c7g8"] Oct 30 00:04:34.970539 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.970449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:34.970609 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:34.970558 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:34.977316 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.977289 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q9h2g"] Oct 30 00:04:34.977432 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.977349 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-msd6l"] Oct 30 00:04:34.977472 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.977437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:34.977597 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:34.977559 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:34.977665 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:34.977651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:34.977753 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:34.977735 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:35.447639 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:35.447607 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:35.549223 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:35.549198 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:37.198632 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:37.198588 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:37.199101 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:37.198672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:37.199101 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:37.198693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:37.199101 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:37.198797 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:37.199101 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:37.198910 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:37.199101 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:37.198969 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:37.976327 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:37.976293 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:37.976480 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:37.976444 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:37.976890 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:37.976872 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:38.002437 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:38.002382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:38.102251 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:38.102204 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:39.198122 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:39.198088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:39.198543 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:39.198195 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:39.198543 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:39.198205 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msd6l" podUID="d080e6f8-404d-4b58-9f16-c4825e28a3c0" Oct 30 00:04:39.198543 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:39.198284 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:39.198543 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:39.198288 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q9h2g" podUID="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" Oct 30 00:04:39.198543 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:39.198350 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c7g8" podUID="97b31541-b41e-482e-bda2-afb18784d5d4" Oct 30 00:04:40.408274 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:40.408237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:40.506629 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:40.506590 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:40.805596 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:40.805535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:40.915087 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:40.915058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:41.023359 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.023324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:41.023618 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.023605 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeReady" Oct 30 00:04:41.023731 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.023722 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:04:41.078228 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.078154 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n2bm4"] Oct 30 00:04:41.080051 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.080030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.081655 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.081628 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wrvj9"] Oct 30 00:04:41.083714 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.083695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.095165 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.095148 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:04:41.100340 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.100322 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:04:41.100798 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.100785 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:04:41.100848 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.100809 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vr82\"" Oct 30 00:04:41.107493 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.107470 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2mrhd"] Oct 30 00:04:41.109924 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.109905 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrvj9"] Oct 30 00:04:41.110013 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.109941 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2bm4"] Oct 30 00:04:41.110072 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.110027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.127458 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.127438 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2mrhd"] Oct 30 00:04:41.136478 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.136454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:41.191123 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.191240 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5996d3a1-c3de-4092-ba1d-71f041e7c37b-data-volume\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.191240 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgb6t\" (UniqueName: \"kubernetes.io/projected/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-api-access-fgb6t\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.191240 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5996d3a1-c3de-4092-ba1d-71f041e7c37b-crio-socket\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.191363 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.191363 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-config-volume\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.191363 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-tmp-dir\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.191363 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191329 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ba5774d-8b53-4430-be28-ba07235cde9f-cert\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.191363 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.191632 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4pl\" (UniqueName: \"kubernetes.io/projected/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-kube-api-access-4g4pl\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.191632 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.191523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86jfk\" (UniqueName: \"kubernetes.io/projected/9ba5774d-8b53-4430-be28-ba07235cde9f-kube-api-access-86jfk\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.195136 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195113 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:04:41.195264 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195180 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:04:41.195447 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195422 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:04:41.195612 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195446 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:04:41.195612 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195501 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:04:41.195612 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195536 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:04:41.195792 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195676 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqlwc\"" Oct 30 00:04:41.195844 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.195806 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntcgl\"" Oct 30 00:04:41.198201 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.198174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:41.198201 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.198203 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:41.198357 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.198218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:41.244507 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.244479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:41.292831 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.292795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ba5774d-8b53-4430-be28-ba07235cde9f-cert\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.292831 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.292832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.293004 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.292864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4pl\" (UniqueName: \"kubernetes.io/projected/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-kube-api-access-4g4pl\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.293004 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.292888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86jfk\" (UniqueName: \"kubernetes.io/projected/9ba5774d-8b53-4430-be28-ba07235cde9f-kube-api-access-86jfk\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.293004 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.292940 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:41.293150 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.293021 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.793002144 +0000 UTC m=+30.061143572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : configmap "kube-rbac-proxy" not found Oct 30 00:04:41.293150 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.293150 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5996d3a1-c3de-4092-ba1d-71f041e7c37b-data-volume\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.293150 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgb6t\" (UniqueName: \"kubernetes.io/projected/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-api-access-fgb6t\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.293336 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.293198 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:41.293336 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.293253 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls podName:8f828c6e-3ce7-4457-b47a-fd34e484f0cd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.793231948 +0000 UTC m=+30.061373367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls") pod "dns-default-n2bm4" (UID: "8f828c6e-3ce7-4457-b47a-fd34e484f0cd") : secret "dns-default-metrics-tls" not found Oct 30 00:04:41.293336 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5996d3a1-c3de-4092-ba1d-71f041e7c37b-crio-socket\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.293336 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.293336 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-config-volume\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.293557 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-tmp-dir\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.293557 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293365 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5996d3a1-c3de-4092-ba1d-71f041e7c37b-crio-socket\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.293690 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.293670 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:41.293740 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293691 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5996d3a1-c3de-4092-ba1d-71f041e7c37b-data-volume\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.293740 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.293696 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-tmp-dir\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.293740 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.293715 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.793702489 +0000 UTC m=+30.061843908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:41.294100 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.294083 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-config-volume\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.295697 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.295676 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:04:41.295978 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.295957 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thx59\"" Oct 30 00:04:41.295978 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.295969 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:04:41.296098 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.295961 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:04:41.296098 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.295969 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pxfs7\"" Oct 30 00:04:41.296098 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.296000 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m7wf2\"" Oct 30 00:04:41.296098 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.296078 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:04:41.297169 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.297121 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ba5774d-8b53-4430-be28-ba07235cde9f-cert\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.315920 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.315889 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4pl\" (UniqueName: \"kubernetes.io/projected/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-kube-api-access-4g4pl\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.315998 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.315892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgb6t\" (UniqueName: \"kubernetes.io/projected/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-api-access-fgb6t\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.316071 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.316057 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86jfk\" (UniqueName: \"kubernetes.io/projected/9ba5774d-8b53-4430-be28-ba07235cde9f-kube-api-access-86jfk\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.344495 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.344468 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:41.396620 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.396598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:41.703503 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.703469 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrvj9"] Oct 30 00:04:41.709092 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:41.709065 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba5774d_8b53_4430_be28_ba07235cde9f.slice/crio-4e7f37311e723c5d3e592bac8687ac0d79789fb8478055781ade88235ba227c8 WatchSource:0}: Error finding container 4e7f37311e723c5d3e592bac8687ac0d79789fb8478055781ade88235ba227c8: Status 404 returned error can't find the container with id 4e7f37311e723c5d3e592bac8687ac0d79789fb8478055781ade88235ba227c8 Oct 30 00:04:41.780090 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.780054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:41.797120 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.797093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:41.797248 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.797146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.797248 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.797182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:41.797248 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.797240 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:41.797396 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.797297 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls podName:8f828c6e-3ce7-4457-b47a-fd34e484f0cd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.79728403 +0000 UTC m=+31.065425448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls") pod "dns-default-n2bm4" (UID: "8f828c6e-3ce7-4457-b47a-fd34e484f0cd") : secret "dns-default-metrics-tls" not found Oct 30 00:04:41.797396 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.797325 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:41.797396 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.797359 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:41.797506 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.797396 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.797376607 +0000 UTC m=+31.065518029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:41.797506 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:41.797417 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.797407012 +0000 UTC m=+31.065548422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : configmap "kube-rbac-proxy" not found Oct 30 00:04:41.881903 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.881873 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:41.989470 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:41.989401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:42.202843 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.202813 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x9kkv"] Oct 30 00:04:42.222447 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.222420 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x9kkv"] Oct 30 00:04:42.222615 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.222514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.236745 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.236722 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 00:04:42.236937 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.236920 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 00:04:42.237045 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.236949 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 00:04:42.237045 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.236955 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 00:04:42.237045 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.237018 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-whzhw\"" Oct 30 00:04:42.300546 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.300404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwl89\" (UniqueName: \"kubernetes.io/projected/9046ee60-4e65-48c9-ab07-21171af6ce8d-kube-api-access-nwl89\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.300546 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.300462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.300886 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.300607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-key\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.401810 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.401779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwl89\" (UniqueName: \"kubernetes.io/projected/9046ee60-4e65-48c9-ab07-21171af6ce8d-kube-api-access-nwl89\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.401964 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.401864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.401964 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.401934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-key\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.402601 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.402565 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.404704 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.404678 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-key\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.424843 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.424815 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwl89\" (UniqueName: \"kubernetes.io/projected/9046ee60-4e65-48c9-ab07-21171af6ce8d-kube-api-access-nwl89\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.458021 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.457978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:42.546933 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.546904 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:42.569653 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.569617 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrvj9" event={"ID":"9ba5774d-8b53-4430-be28-ba07235cde9f","Type":"ContainerStarted","Data":"4e7f37311e723c5d3e592bac8687ac0d79789fb8478055781ade88235ba227c8"} Oct 30 00:04:42.805869 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.805829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.805909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.805951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:42.805959 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:42.806042 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls podName:8f828c6e-3ce7-4457-b47a-fd34e484f0cd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.806016211 +0000 UTC m=+33.074157633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls") pod "dns-default-n2bm4" (UID: "8f828c6e-3ce7-4457-b47a-fd34e484f0cd") : secret "dns-default-metrics-tls" not found Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:42.806059 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:42.806109 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.806097465 +0000 UTC m=+33.074238884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : configmap "kube-rbac-proxy" not found Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:42.806059 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:42.806337 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:42.806137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.806131657 +0000 UTC m=+33.074273102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:42.869629 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:42.869541 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x9kkv"] Oct 30 00:04:42.874592 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:42.874546 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9046ee60_4e65_48c9_ab07_21171af6ce8d.slice/crio-d8172ab1e3a202c9af4d4fa4ebc1c0203004afa48a42dbc863f898e7c49ccbdb WatchSource:0}: Error finding container d8172ab1e3a202c9af4d4fa4ebc1c0203004afa48a42dbc863f898e7c49ccbdb: Status 404 returned error can't find the container with id d8172ab1e3a202c9af4d4fa4ebc1c0203004afa48a42dbc863f898e7c49ccbdb Oct 30 00:04:43.190225 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:43.190151 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:43.574092 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:43.574052 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" event={"ID":"9046ee60-4e65-48c9-ab07-21171af6ce8d","Type":"ContainerStarted","Data":"d8172ab1e3a202c9af4d4fa4ebc1c0203004afa48a42dbc863f898e7c49ccbdb"} Oct 30 00:04:44.276551 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.276522 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8vxtm_8aad4aae-3822-4a95-aa08-ecc8fcce7bee/dns-node-resolver/0.log" Oct 30 00:04:44.580601 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.577958 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrvj9" event={"ID":"9ba5774d-8b53-4430-be28-ba07235cde9f","Type":"ContainerStarted","Data":"e642abf831268096f9bb01e98f9586b72b67e3761ebe5799623b9e4bb7cfe485"} Oct 30 00:04:44.627330 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.627273 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wrvj9" podStartSLOduration=0.855715666 podStartE2EDuration="3.627259074s" totalImagesPullingTime="2.771543408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:44.626991131 +0000 UTC m=+32.895132558" watchObservedRunningTime="2026-10-30 00:04:44.627259074 +0000 UTC m=+32.895400500" Oct 30 00:04:44.820089 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.819996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:44.820089 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.820081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:44.820306 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.820132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:44.820306 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.820082 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:44.820306 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.820237 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.820214969 +0000 UTC m=+37.088356388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : configmap "kube-rbac-proxy" not found Oct 30 00:04:44.820306 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.820242 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:44.820306 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.820306 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls podName:8f828c6e-3ce7-4457-b47a-fd34e484f0cd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.820287344 +0000 UTC m=+37.088428762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls") pod "dns-default-n2bm4" (UID: "8f828c6e-3ce7-4457-b47a-fd34e484f0cd") : secret "dns-default-metrics-tls" not found Oct 30 00:04:44.820560 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.820239 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:44.820560 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.820357 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.820344857 +0000 UTC m=+37.088486279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:44.920946 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.920907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:44.921130 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.920981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:44.921130 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.921062 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:44.921246 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:44.921141 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:05:16.921122811 +0000 UTC m=+65.189264215 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : secret "metrics-daemon-secret" not found Oct 30 00:04:44.923628 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:44.923597 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:45.022132 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.022093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:45.025076 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.025047 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:45.113628 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.113539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:45.130306 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.130251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:45.437698 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.437621 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q9h2g"] Oct 30 00:04:45.456991 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.456949 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2c7g8"] Oct 30 00:04:45.476683 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.476664 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q84nm_557dfb83-6842-4e18-b8e2-3426d7ba17ed/node-ca/0.log" Oct 30 00:04:45.503698 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:45.503673 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b4ad3c_8956_4e4a_a68f_2df514e7dc25.slice/crio-88f56a87e0e15260bf19f3d9fa3e832ec93a3ba9ed2f2d447ad699574c22dbc7 WatchSource:0}: Error finding container 88f56a87e0e15260bf19f3d9fa3e832ec93a3ba9ed2f2d447ad699574c22dbc7: Status 404 returned error can't find the container with id 88f56a87e0e15260bf19f3d9fa3e832ec93a3ba9ed2f2d447ad699574c22dbc7 Oct 30 00:04:45.504402 ip-10-0-138-81 kubenswrapper[2580]: W1030 00:04:45.504382 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b31541_b41e_482e_bda2_afb18784d5d4.slice/crio-5a4e18ef203557b1c52f71955422fd8d375eef5c16cac6aac2e0862ecbaf2beb WatchSource:0}: Error finding container 5a4e18ef203557b1c52f71955422fd8d375eef5c16cac6aac2e0862ecbaf2beb: Status 404 returned error can't find the container with id 5a4e18ef203557b1c52f71955422fd8d375eef5c16cac6aac2e0862ecbaf2beb Oct 30 00:04:45.580922 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.580896 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2c7g8" event={"ID":"97b31541-b41e-482e-bda2-afb18784d5d4","Type":"ContainerStarted","Data":"5a4e18ef203557b1c52f71955422fd8d375eef5c16cac6aac2e0862ecbaf2beb"} Oct 30 00:04:45.582076 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.582052 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q9h2g" event={"ID":"26b4ad3c-8956-4e4a-a68f-2df514e7dc25","Type":"ContainerStarted","Data":"88f56a87e0e15260bf19f3d9fa3e832ec93a3ba9ed2f2d447ad699574c22dbc7"} Oct 30 00:04:45.644844 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.644817 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:45.796802 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:45.796726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:46.475881 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:46.475852 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wrvj9_9ba5774d-8b53-4430-be28-ba07235cde9f/serve-healthcheck-canary/0.log" Oct 30 00:04:46.590656 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:46.590614 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" event={"ID":"9046ee60-4e65-48c9-ab07-21171af6ce8d","Type":"ContainerStarted","Data":"d548c5e2d6e2f661cade52387f82a981fcc86dc928dcbfa6fdff9dc27869d452"} Oct 30 00:04:46.629917 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:46.629861 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" podStartSLOduration=1.979549808 podStartE2EDuration="4.629843279s" totalImagesPullingTime="2.650293471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:46.629362027 +0000 UTC m=+34.897503456" watchObservedRunningTime="2026-10-30 00:04:46.629843279 +0000 UTC m=+34.897984719" Oct 30 00:04:46.690921 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:46.690880 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:46.916168 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:46.916136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:47.019345 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:47.017442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:47.120894 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:47.120863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:47.224317 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:47.224247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:47.452394 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:47.452359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:47.552761 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:47.552680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:47.851749 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:47.851719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:48.653689 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:48.653650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:48.852493 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:48.852454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:48.852690 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:48.852508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:48.852690 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:48.852535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:48.852690 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:48.852636 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:48.852690 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:48.852677 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:48.852902 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:48.852708 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls podName:8f828c6e-3ce7-4457-b47a-fd34e484f0cd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:56.85269135 +0000 UTC m=+45.120832776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls") pod "dns-default-n2bm4" (UID: "8f828c6e-3ce7-4457-b47a-fd34e484f0cd") : secret "dns-default-metrics-tls" not found Oct 30 00:04:48.852902 ip-10-0-138-81 kubenswrapper[2580]: E1030 00:04:48.852740 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:56.852724482 +0000 UTC m=+45.120865890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:48.853042 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:48.853014 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:49.052079 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.051953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:49.253870 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.253841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:49.452469 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.452427 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:49.555355 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.555323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:49.600477 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.600443 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q9h2g" event={"ID":"26b4ad3c-8956-4e4a-a68f-2df514e7dc25","Type":"ContainerStarted","Data":"11c89cd96592ba9363f8e65e961d60b4185ba0e454e4cf2044409fc5bf07d057"} Oct 30 00:04:49.601157 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.601139 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:49.639799 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.639758 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q9h2g" podStartSLOduration=34.000227198 podStartE2EDuration="37.639744411s" totalImagesPullingTime="3.639517213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:49.639093101 +0000 UTC m=+37.907234528" watchObservedRunningTime="2026-10-30 00:04:49.639744411 +0000 UTC m=+37.907885831" Oct 30 00:04:49.852095 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:49.852055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:50.447482 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:50.447446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:50.548251 ip-10-0-138-81 kubenswrapper[2580]: I1030 00:04:50.548224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:51.002866 ip-10-0-138-81 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:04:51.015957 ip-10-0-138-81 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:04:51.016121 ip-10-0-138-81 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:04:51.016167 ip-10-0-138-81 systemd[1]: kubelet.service: Consumed 2.305s CPU time. Oct 30 00:04:51.030151 ip-10-0-138-81 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:04:51.094921 ip-10-0-138-81 kubenswrapper[4433]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:51.094921 ip-10-0-138-81 kubenswrapper[4433]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:04:51.094921 ip-10-0-138-81 kubenswrapper[4433]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:51.094921 ip-10-0-138-81 kubenswrapper[4433]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:51.097299 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097214 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.097299 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097294 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.097299 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097300 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097307 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097314 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097319 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097323 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097327 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097331 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097334 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097338 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097342 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097346 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097350 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097355 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097359 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097363 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097368 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097372 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097376 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097380 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.097526 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097384 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097388 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097392 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097395 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097399 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097403 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097407 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097411 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097415 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097420 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097424 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097427 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097432 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097437 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097442 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097446 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097451 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097455 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097459 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097463 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.098386 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097468 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097474 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097478 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097483 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097488 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097492 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097497 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097502 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097506 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097511 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097515 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097520 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097524 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097528 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097532 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097536 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097540 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097545 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097550 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.099295 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097555 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097559 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097564 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097586 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097592 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097596 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097600 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097604 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097608 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097613 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097617 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097621 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097625 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097629 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097633 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097640 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097646 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097650 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097655 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.100296 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097659 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097665 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097669 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097673 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097677 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097681 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097687 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097691 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097695 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097699 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097704 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097709 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097713 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097718 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097722 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097726 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097730 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097735 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097739 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097743 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.101182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097747 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097752 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097759 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097763 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097767 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097771 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.097776 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099020 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099031 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099035 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099040 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099044 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099049 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099053 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099058 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099062 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099067 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099071 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099076 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.102114 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099080 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099085 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099090 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099095 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099100 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099107 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099114 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099119 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099124 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099129 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099134 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099138 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099143 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099147 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099151 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099156 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099160 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099164 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099168 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099173 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.103018 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099177 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099182 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099186 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099190 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099194 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099199 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099203 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099207 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099211 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099217 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099221 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099225 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099229 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099236 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099242 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099248 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099253 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099258 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099262 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.103809 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099267 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099271 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099276 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099281 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099285 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099290 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099294 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099298 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099302 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099306 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099310 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099315 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099319 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099323 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099328 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099332 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099337 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099341 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099345 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099350 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.104297 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099354 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099358 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099371 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099376 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099381 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099385 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099390 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099394 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099398 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099402 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099406 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099410 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099414 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099418 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099423 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099427 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099431 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099435 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099439 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.104814 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099443 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099447 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099452 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099456 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099460 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099464 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099469 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099473 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099477 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099481 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099486 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099495 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099499 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099503 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099508 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.099513 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100374 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100384 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100389 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.105339 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100396 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100402 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100407 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100411 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100416 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100421 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100425 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100429 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100433 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100438 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100442 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100446 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100450 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100455 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100459 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100463 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100468 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100472 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100476 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100481 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.105834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100485 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100489 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100493 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100498 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100505 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100510 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100515 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100519 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100523 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100528 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100532 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100536 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100541 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100545 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100549 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100554 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100558 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100562 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100567 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100587 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.106328 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100593 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100597 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100600 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100605 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100609 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100613 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100618 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100622 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100628 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100632 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100636 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100640 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100644 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100648 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100653 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100657 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100663 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100668 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100672 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100676 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.106852 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100680 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100684 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100689 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100694 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100698 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100702 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100706 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100710 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100715 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100719 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100723 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100728 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100732 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100736 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100740 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100744 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100748 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100753 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100757 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100761 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.107673 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100768 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100772 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100776 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100780 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100784 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100789 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100793 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100797 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100803 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100808 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100812 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100816 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100820 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100824 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100829 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100833 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100838 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100842 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100846 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.108202 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100850 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100857 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100862 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.100867 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101009 4433 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101026 4433 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101034 4433 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101041 4433 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101048 4433 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101053 4433 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101060 4433 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101067 4433 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101072 4433 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101080 4433 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101086 4433 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101091 4433 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101096 4433 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101101 4433 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101105 4433 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101110 4433 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101115 4433 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101122 4433 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101129 4433 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:04:51.108722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101134 4433 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101138 4433 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101144 4433 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101150 4433 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101156 4433 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101161 4433 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101166 4433 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101172 4433 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101177 4433 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101181 4433 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101186 4433 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101190 4433 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101195 4433 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101201 4433 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101206 4433 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101212 4433 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101217 4433 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101221 4433 flags.go:64] FLAG: --enable-server="true" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101226 4433 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101233 4433 flags.go:64] FLAG: --event-burst="100" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101239 4433 flags.go:64] FLAG: --event-qps="50" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101243 4433 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101250 4433 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101255 4433 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:04:51.109273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101261 4433 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101266 4433 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101271 4433 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101276 4433 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101281 4433 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101286 4433 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101292 4433 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101297 4433 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101302 4433 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101307 4433 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101312 4433 flags.go:64] FLAG: --feature-gates="" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101319 4433 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101324 4433 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101329 4433 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101334 4433 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101340 4433 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101345 4433 flags.go:64] FLAG: --help="false" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101350 4433 flags.go:64] FLAG: --hostname-override="ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101355 4433 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101360 4433 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101365 4433 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101371 4433 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101377 4433 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101381 4433 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:04:51.109877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101386 4433 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101390 4433 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101395 4433 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101400 4433 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101405 4433 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101410 4433 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101417 4433 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101422 4433 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101427 4433 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101432 4433 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101436 4433 flags.go:64] FLAG: --lock-file="" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101441 4433 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101446 4433 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101451 4433 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101462 4433 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101467 4433 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101472 4433 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101477 4433 flags.go:64] FLAG: --logging-format="text" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101482 4433 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101487 4433 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101491 4433 flags.go:64] FLAG: --manifest-url="" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101496 4433 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101503 4433 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101508 4433 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101515 4433 flags.go:64] FLAG: --max-pods="110" Oct 30 00:04:51.110592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101519 4433 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101524 4433 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101529 4433 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101534 4433 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101539 4433 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101544 4433 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101549 4433 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101559 4433 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101564 4433 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101585 4433 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101591 4433 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101595 4433 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101600 4433 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101608 4433 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101612 4433 flags.go:64] FLAG: --port="10250" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101617 4433 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101622 4433 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01eb03f0cf5adc396" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101627 4433 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101632 4433 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101638 4433 flags.go:64] FLAG: --register-node="true" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101642 4433 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101651 4433 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101656 4433 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101660 4433 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101665 4433 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101671 4433 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:04:51.111216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101676 4433 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101681 4433 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101685 4433 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101690 4433 flags.go:64] FLAG: --runonce="false" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101695 4433 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101700 4433 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101706 4433 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101711 4433 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101715 4433 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101721 4433 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101725 4433 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101730 4433 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101735 4433 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101740 4433 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101745 4433 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101750 4433 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101755 4433 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101760 4433 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101764 4433 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101775 4433 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101780 4433 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101784 4433 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101791 4433 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101796 4433 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101801 4433 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:04:51.111891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101806 4433 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101811 4433 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101817 4433 flags.go:64] FLAG: --v="2" Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101824 4433 flags.go:64] FLAG: --version="false" Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101831 4433 flags.go:64] FLAG: --vmodule="" Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101837 4433 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.101843 4433 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102062 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102069 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102074 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102079 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102084 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102089 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102093 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102099 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102103 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102108 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102112 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102116 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102122 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102129 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.112508 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102134 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102139 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102143 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102148 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102152 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102158 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102163 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102167 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102171 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102176 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102180 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102184 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102189 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102195 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102199 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102204 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102208 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102212 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102217 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102221 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.113125 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102225 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102229 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102234 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102238 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102242 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102246 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102252 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102256 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102260 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102264 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102269 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102273 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102277 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102282 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102287 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102291 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102295 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102302 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102306 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.113624 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102311 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102315 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102319 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102324 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102328 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102333 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102339 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102343 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102348 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102352 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102356 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102360 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102364 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102369 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102373 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102377 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102382 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102386 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102390 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.114111 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102394 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102399 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102403 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102408 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102412 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102417 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102421 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102425 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102429 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102434 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102439 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102444 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102448 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102452 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102457 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102461 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102466 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102470 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102476 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.114596 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102484 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102489 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102495 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102499 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102503 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102508 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102512 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102516 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102521 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102525 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102529 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102533 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102537 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102541 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102546 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.102604 4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:51.115118 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102661 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102668 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102673 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102678 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102683 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102687 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102691 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102695 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102702 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102706 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102711 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102717 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102723 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102728 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102732 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102736 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102740 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102746 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102750 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.115651 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102755 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102759 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102763 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102768 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102772 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102776 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102780 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102785 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102789 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102793 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102797 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102802 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102807 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102812 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102816 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102821 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102825 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102829 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102833 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.116221 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102838 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102844 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102852 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102865 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102872 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102877 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102881 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102885 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102890 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102894 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102898 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102904 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102909 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102913 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102917 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102921 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102925 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102929 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102934 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102938 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.116708 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102943 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102947 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102951 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102955 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102960 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102964 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102968 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102973 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102977 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102982 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102986 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102990 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102994 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.102999 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103005 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103009 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103014 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103018 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103022 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103026 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.117405 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103030 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103034 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103038 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103043 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103047 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103051 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103055 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103060 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103064 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103069 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103073 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103078 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103082 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103086 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103090 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103094 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103098 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103103 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103107 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.117912 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103111 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103116 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103120 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103125 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103129 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103135 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103138 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103145 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.103149 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.107338 4433 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.107357 4433 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.107381 4433 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.107387 4433 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108131 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108142 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108146 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108150 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108154 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108157 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.118369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108160 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108163 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108166 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108169 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108171 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108174 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108177 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108180 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108182 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108185 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108188 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108191 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108193 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108196 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108199 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108201 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108204 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108209 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108213 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.118837 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108216 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108220 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108224 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108227 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108230 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108232 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108235 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108240 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108243 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108246 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108248 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108251 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108254 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108257 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108259 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108262 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108265 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108268 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108270 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108273 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.119338 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108276 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108280 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108283 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108285 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108288 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108291 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108293 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108297 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108299 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108302 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108305 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108308 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108310 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108314 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108318 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108321 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108325 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108328 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108330 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.119929 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108333 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108335 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108338 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108341 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108344 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108346 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108349 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108351 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108354 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108357 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108359 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108362 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108365 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108367 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108370 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108373 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108376 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108379 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108381 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108384 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.120377 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108387 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108389 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108392 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108395 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108403 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108406 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108409 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108411 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108414 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108417 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108419 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108422 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108425 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108427 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108430 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108434 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108436 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108439 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108442 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108444 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.120884 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108447 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108449 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.108479 4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108634 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108641 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108644 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108647 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108651 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108656 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108659 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108662 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108665 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108668 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108671 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108674 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108677 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.121369 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108680 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108682 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108685 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108688 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108690 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108693 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108696 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108698 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108701 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108703 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108706 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108709 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108711 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108714 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108716 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108719 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108721 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108724 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108727 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108730 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.121776 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108732 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108735 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108738 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108740 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108743 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108746 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108748 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108751 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108753 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108756 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108759 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108761 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108764 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108766 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108769 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108772 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108774 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108777 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108779 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.122253 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108782 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108784 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108787 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108790 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108792 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108795 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108798 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108800 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108803 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108805 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108808 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108810 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108813 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108815 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108818 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108821 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108823 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108826 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108828 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108831 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.122724 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108834 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108837 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108839 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108842 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108844 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108846 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108849 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108852 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108854 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108857 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108860 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108862 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108865 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108867 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108870 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108872 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108875 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108878 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108880 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108883 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.123205 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108885 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108888 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108890 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108893 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108895 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108898 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108900 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108903 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108906 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108909 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108914 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108917 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108921 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108925 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.108947 4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108978 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:51.123687 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108982 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108985 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108989 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108991 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108994 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108997 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.108999 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109002 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109004 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109007 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109010 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109012 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109015 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109017 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109020 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109022 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109025 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109027 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109030 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:51.124101 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109032 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109038 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109041 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109044 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109047 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109049 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109052 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109055 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109057 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109060 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109063 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109065 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109068 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109070 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109073 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109076 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109078 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109081 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109083 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:51.124826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109086 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109089 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109091 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109094 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109096 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109099 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109101 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109104 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109106 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109109 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109111 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109114 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109117 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109119 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109123 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109126 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109128 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109131 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109134 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:51.125547 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109136 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109140 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109144 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109147 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109150 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109153 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109155 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109158 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109161 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109163 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109167 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109171 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109174 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109177 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109180 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109184 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109186 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109189 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109192 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:51.126477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109195 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109197 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109200 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109202 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109205 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109208 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109210 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109213 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109216 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109219 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109222 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109224 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109228 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109231 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109234 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109236 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109239 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109241 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109244 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109248 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:51.127263 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109251 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109253 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109256 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109258 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109261 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109263 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109266 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109268 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:51.109271 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.109327 4433 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.110273 4433 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.110318 4433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.110503 4433 server.go:1014] "Starting client certificate rotation" Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.110606 4433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.110656 4433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:59:12 +0000 UTC" deadline="2028-06-08 19:04:06.500232088 +0000 UTC" Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.110691 4433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14106h59m15.389543859s" Oct 30 00:04:51.128130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.111031 4433 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:51.128873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.112124 4433 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:51.128873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.113517 4433 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:04:51.128873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.115017 4433 log.go:25] "Validated CRI v1 image API" Oct 30 00:04:51.128873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.115799 4433 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:04:51.128873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.117211 4433 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c95b7c3d-9589-41d6-ba9b-9d3e0a1a4c7b:/dev/nvme0n1p4 dc20bbbe-f42b-4ee0-bcc4-a1a4a28b499e:/dev/nvme0n1p3] Oct 30 00:04:51.129107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.117230 4433 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02d0693d91f8a2f25f45c9c9a1671c2d61baf7a3e85c5a7319f4bd7e3e76f683/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02d0693d91f8a2f25f45c9c9a1671c2d61baf7a3e85c5a7319f4bd7e3e76f683/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/066c3492bb0be8b1cbbdb58a243bb56bab09bebe0654145d1598ad774576ab25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/066c3492bb0be8b1cbbdb58a243bb56bab09bebe0654145d1598ad774576ab25/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d199ecc8c554cb0c1ba0bf52821e25cbbc9d4414730154e3b6041445cab9610/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d199ecc8c554cb0c1ba0bf52821e25cbbc9d4414730154e3b6041445cab9610/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42cffa649dd705f1bd9819b5d8bee55e2be2b5356657d7762a529ea006da3ec0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42cffa649dd705f1bd9819b5d8bee55e2be2b5356657d7762a529ea006da3ec0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e7f37311e723c5d3e592bac8687ac0d79789fb8478055781ade88235ba227c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e7f37311e723c5d3e592bac8687ac0d79789fb8478055781ade88235ba227c8/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57c3db016636eaf05d5c01bc874808cf8e1c94d924bf6955af1127c11b992ca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57c3db016636eaf05d5c01bc874808cf8e1c94d924bf6955af1127c11b992ca7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59da8546fed618e7a8bdbb760c4edf13232e4ee575caec822fbbae1811c38ca0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59da8546fed618e7a8bdbb760c4edf13232e4ee575caec822fbbae1811c38ca0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a4e18ef203557b1c52f71955422fd8d375eef5c16cac6aac2e0862ecbaf2beb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a4e18ef203557b1c52f71955422fd8d375eef5c16cac6aac2e0862ecbaf2beb/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88f56a87e0e15260bf19f3d9fa3e832ec93a3ba9ed2f2d447ad699574c22dbc7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88f56a87e0e15260bf19f3d9fa3e832ec93a3ba9ed2f2d447ad699574c22dbc7/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7bb1ae049df5fd6639e4de1fea18dd7e1517dd7224cc2ed155ddfe71ab9d424/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7bb1ae049df5fd6639e4de1fea18dd7e1517dd7224cc2ed155ddfe71ab9d424/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8172ab1e3a202c9af4d4fa4ebc1c0203004afa48a42dbc863f898e7c49ccbdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8172ab1e3a202c9af4d4fa4ebc1c0203004afa48a42dbc863f898e7c49ccbdb/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f78ce33c22222453b260ca7cab74c28787ed573b92cefd4f084db0715f02e9e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f78ce33c22222453b260ca7cab74c28787ed573b92cefd4f084db0715f02e9e5/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volumes/kubernetes.io~projected/kube-api-access-nww5d:{mountpoint:/var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volumes/kubernetes.io~projected/kube-api-access-nww5d major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26b4ad3c-8956-4e4a-a68f-2df514e7dc25/volumes/kubernetes.io~projected/kube-api-access-872tn:{mountpoint:/var/lib/kubelet/pods/26b4ad3c-8956-4e4a-a68f-2df514e7dc25/volumes/kubernetes.io~projected/kube-api-access-872tn major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/557dfb83-6842-4e18-b8e2-3426d7ba17ed/volumes/kubernetes.io~projected/kube-api-access-kc54t:{mountpoint:/var/lib/kubelet/pods/557dfb83-6842-4e18-b8e2-3426d7ba17ed/volumes/kubernetes.io~projected/kube-api-access-kc54t major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5996d3a1-c3de-4092-ba1d-71f041e7c37b/volumes/kubernetes.io~projected/kube-api-access-fgb6t:{mountpoint:/var/lib/kubelet/pods/5996d3a1-c3de-4092-ba1d-71f041e7c37b/volumes/kubernetes.io~projected/kube-api-access-fgb6t major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/679a9efd-b724-4566-91ac-ba5000919e94/volumes/kubernetes.io~projected/kube-api-access-bhmkp:{mountpoint:/var/lib/kubelet/pods/679a9efd-b724-4566-91ac-ba5000919e94/volumes/kubernetes.io~projected/kube-api-access-bhmkp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/692f7a39-6f14-4a82-9990-60fd90fba46b/volumes/kubernetes.io~projected/kube-api-access-zttcl:{mountpoint:/var/lib/kubelet/pods/692f7a39-6f14-4a82-9990-60fd90fba46b/volumes/kubernetes.io~projected/kube-api-access-zttcl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aad4aae-3822-4a95-aa08-ecc8fcce7bee/volumes/kubernetes.io~projected/kube-api-access-dwjgg:{mountpoint:/var/lib/kubelet/pods/8aad4aae-3822-4a95-aa08-ecc8fcce7bee/volumes/kubernetes.io~projected/kube-api-access-dwjgg major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~projected/kube-api-access-cxl8s:{mountpoint:/var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~projected/kube-api-access-cxl8s major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f828c6e-3ce7-4457-b47a-fd34e484f0cd/volumes/kubernetes.io~projected/kube-api-access-4g4pl:{mountpoint:/var/lib/kubelet/pods/8f828c6e-3ce7-4457-b47a-fd34e484f0cd/volumes/kubernetes.io~projected/kube-api-access-4g4pl major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9046ee60-4e65-48c9-ab07-21171af6ce8d/volumes/kubernetes.io~projected/kube-api-access-nwl89:{mountpoint:/var/lib/kubelet/pods/9046ee60-4e65-48c9-ab07-21171af6ce8d/volumes/kubernetes.io~projected/kube-api-access-nwl89 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9046ee60-4e65-48c9-ab07-21171af6ce8d/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/9046ee60-4e65-48c9-ab07-21171af6ce8d/volumes/kubernetes.io~secret/signing-key major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97b31541-b41e-482e-bda2-afb18784d5d4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/97b31541-b41e-482e-bda2-afb18784d5d4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ba5774d-8b53-4430-be28-ba07235cde9f/volumes/kubernetes.io~projected/kube-api-access-86jfk:{mountpoint:/var/lib/kubelet/pods/9ba5774d-8b53-4430-be28-ba07235cde9f/volumes/kubernetes.io~projected/kube-api-access-86jfk major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ba5774d-8b53-4430-be28-ba07235cde9f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9ba5774d-8b53-4430-be28-ba07235cde9f/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a37bd471-b857-4907-b357-aae3315a2721/volumes/kubernetes.io~projected/kube-api-access-29b9r:{mountpoint:/var/lib/kubelet/pods/a37bd471-b857-4907-b357-aae3315a2721/volumes/kubernetes.io~projected/kube-api-access-29b9r major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a79d2269-64f4-444e-9d57-c081a6744d81/volumes/kubernetes.io~projected/kube-api-access-xnvfv:{mountpoint:/var/lib/kubelet/pods/a79d2269-64f4-444e-9d57-c081a6744d81/volumes/kubernetes.io~projected/kube-api-access-xnvfv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d080e6f8-404d-4b58-9f16-c4825e28a3c0/volumes/kubernetes.io~projected/kube-api-access-fqf26:{mountpoint:/var/lib/kubelet/pods/d080e6f8-404d-4b58-9f16-c4825e28a3c0/volumes/kubernetes.io~projected/kube-api-access-fqf26 major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/20bb8e3e2e4dc73f761c5b68ecd88060ea33c9a8812b6a53e30a3ef95a18f7ff/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/428df7aed3d5b8d270dea28639f6b34baf1405785adf994376ceefa9d45f6988/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6d9c37a36fd222e7de019bba570e3eed02e380e0c2c3da9ac8516ca048af3e26/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/47e1e0d455b5855645d28329823949c114c5629017f2b3951950da98d1d1765c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e2494e454dafd5d7c29aaba347239ba696a06486dfa952edb88b401545988be1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/40997a0ce8794c9462ca6aa2ad6089105a97d3454959fa483ac868bbeae53680/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6237d7eab5c99e461441f2fe612c81fe75e4a2d444ee048d4e0a45afe818a0e1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/32fe8010706f5d3bd35a5a169795f6c549f8cac9c8a7c3ee9812c9cbefc8d86b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4f624bdd5e723d64dde542d7f642a3989456f6a662a533d03981242e82c1b1ec/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ff552cbcee4d6f2692a96cfea2df4d0f4bd990294fe8c4c98ac65683d0e18f17/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7283e06588bf301053fe824d29ac6f81f210103587de95316357d688c6f8b1cb/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/304c5378195cb68761d4fbd03c38f151ed7d202228fd1bfcabd8a27fb87e40ae/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/170e7064c8690bfd2e637858a922dd90720aa796ac51ac918341920249af0b06/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/68356e90273abc7241d81e9e51c57dfdeaac91c8818bc69f2fab503a7401fa09/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/30eb858fd95e9864846ca53068ddf99406b4c2a3019d1f151bb79bd92a3c0651/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9fc69699ceca2023c6eda2b9d1684ebcdf9ac8bc2e245ad418a8bb2fd629fabd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c5174a12926bd656182a1b42bcad248278d8ea2c7b78d5d032acff04cd310aff/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e4e4cc7c781e5436486657562ce3fb54736e03af1e1a2682796eef609d5f3403/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8247dbbea8e338b464bce8d66b34f02abf7eb7c1178eeb34e70e89d6cd238b64/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/38658bcdd0e87d91d323274dd359b40a5d0edbab380997855f1ed6da6214ccdd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8394f92055e69e2f41b2c0f4e8806e0c3f2aa9a79c62eff08bcf9031c104dd92/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/92d94428d90c37248361726bececb9fd1f9bd03a7040bdd21b57ec491abd5515/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/68482178046db563cf174bb9acd7377d27affe0e2686e18baa2d7cce250b462c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4eeaa8b8ec3acde8ec5fcc9bebb35f8485dfdbfdeb063bf57879d984aac64617/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b97a363e708302888e8988a220dd45a8b7338523230fa054a6b019caf2cd6d55/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/7fe8cba4a9c2b5296f03b82520158c8f92746c72d6f40be415d8fc45c7f3727d/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/2cced0b57a9d54017bb56e072c7c36e3e71c6f960075a5d56df80f28e11cd473/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/bb62e11571dc6c86d90d46f3f6fbb7c564c589e8c3989c11084176efcfc19bdb/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/a11979cda0cce0426ac730acd9c760c001b09fd54496765b350f9cdf404caf2a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/76c668487d83611450d54e83373ef6e08ee9f180e89522d00111e69d271728fe/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1bc076acf49cfbb5261e6612401053bd0a3050339e89c169009f2e66c0802d13/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1a17b4992fe7ab44b20f70335b10496e609f3030b1d3c20c3a2459abd036b703/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/893dada1b285ae83b87f6bdde4d76ad7c1ac9c650afd99648d26dd3634c472d0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/df42d55347874c3e3e42eba50787bce9d7693e5b2282fbeefaf5063609c61375/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/53b054b34cb30f347661ad3ca34588e93b82896ffd574265cfa4a2ff798ac4ab/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6a603cfa81a1f76cda7deb7182665521ec137bd86df8b3b2025647c8e977e5c7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/c1a89bc3bfe253d959415c449f11838b88e538e16d4f253915de832daf591afa/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/b5386fa58d8b545f669a999d4eadb77de0c7611a0cd9ab79a6004945d0a9134e/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/aa4542809033a5ddb31f699de398030decb7c21920a4e16119914d633a117fa9/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.123521 4433 manager.go:217] Machine: {Timestamp:2026-10-30 00:04:51.123067787 +0000 UTC m=+0.057136317 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104399 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2de017dc812d63e9557b64fce9f674 SystemUUID:ec2de017-dc81-2d63-e955-7b64fce9f674 BootID:96b0af6c-3d85-41b8-8a87-a293bf82c5b8 Filesystems:[{Device:/var/lib/kubelet/pods/9ba5774d-8b53-4430-be28-ba07235cde9f/volumes/kubernetes.io~projected/kube-api-access-86jfk DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f78ce33c22222453b260ca7cab74c28787ed573b92cefd4f084db0715f02e9e5/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volumes/kubernetes.io~projected/kube-api-access-nww5d DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a79d2269-64f4-444e-9d57-c081a6744d81/volumes/kubernetes.io~projected/kube-api-access-xnvfv DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f828c6e-3ce7-4457-b47a-fd34e484f0cd/volumes/kubernetes.io~projected/kube-api-access-4g4pl DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8172ab1e3a202c9af4d4fa4ebc1c0203004afa48a42dbc863f898e7c49ccbdb/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~projected/kube-api-access-cxl8s DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5996d3a1-c3de-4092-ba1d-71f041e7c37b/volumes/kubernetes.io~projected/kube-api-access-fgb6t DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d080e6f8-404d-4b58-9f16-c4825e28a3c0/volumes/kubernetes.io~projected/kube-api-access-fqf26 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/679a9efd-b724-4566-91ac-ba5000919e94/volumes/kubernetes.io~projected/kube-api-access-bhmkp DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d199ecc8c554cb0c1ba0bf52821e25cbbc9d4414730154e3b6041445cab9610/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/066c3492bb0be8b1cbbdb58a243bb56bab09bebe0654145d1598ad774576ab25/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97b31541-b41e-482e-bda2-afb18784d5d4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59da8546fed618e7a8bdbb760c4edf13232e4ee575caec822fbbae1811c38ca0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e7f37311e723c5d3e592bac8687ac0d79789fb8478055781ade88235ba227c8/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42cffa649dd705f1bd9819b5d8bee55e2be2b5356657d7762a529ea006da3ec0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9ba5774d-8b53-4430-be28-ba07235cde9f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9046ee60-4e65-48c9-ab07-21171af6ce8d/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:207 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8aad4aae-3822-4a95-aa08-ecc8fcce7bee/volumes/kubernetes.io~projected/kube-api-access-dwjgg DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a37bd471-b857-4907-b357-aae3315a2721/volumes/kubernetes.io~projected/kube-api-access-29b9r DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88f56a87e0e15260bf19f3d9fa3e832ec93a3ba9ed2f2d447ad699574c22dbc7/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a4e18ef203557b1c52f71955422fd8d375eef5c16cac6aac2e0862ecbaf2beb/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8ce3b143-5c8f-4170-8f84-095f7668c53d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/692f7a39-6f14-4a82-9990-60fd90fba46b/volumes/kubernetes.io~projected/kube-api-access-zttcl DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9046ee60-4e65-48c9-ab07-21171af6ce8d/volumes/kubernetes.io~projected/kube-api-access-nwl89 DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26b4ad3c-8956-4e4a-a68f-2df514e7dc25/volumes/kubernetes.io~projected/kube-api-access-872tn DeviceMajor:0 DeviceMinor:220 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02d0693d91f8a2f25f45c9c9a1671c2d61baf7a3e85c5a7319f4bd7e3e76f683/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57c3db016636eaf05d5c01bc874808cf8e1c94d924bf6955af1127c11b992ca7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/557dfb83-6842-4e18-b8e2-3426d7ba17ed/volumes/kubernetes.io~projected/kube-api-access-kc54t DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/079ae719-a70f-4d46-923c-aeb73ce07125/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7bb1ae049df5fd6639e4de1fea18dd7e1517dd7224cc2ed155ddfe71ab9d424/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4e7f37311e723c5 MacAddress:ba:3a:36:7a:20:63 Speed:10000 Mtu:8901} {Name:5a4e18ef203557b MacAddress:f2:a6:c7:da:9a:44 Speed:10000 Mtu:8901} {Name:88f56a87e0e1526 MacAddress:8a:49:26:cf:d5:c8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c1:82:8e:05 Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:0e:99:7a:69:2a Speed:0 Mtu:8901} {Name:d8172ab1e3a202c MacAddress:36:b5:fc:c0:0b:c0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c1:82:8e:05 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:be:b8:dc:28:21:78 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:26:97:78:24:8a:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.123760 4433 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.123802 4433 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124041 4433 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124064 4433 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124511 4433 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124526 4433 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124541 4433 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124568 4433 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124705 4433 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.124988 4433 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.125143 4433 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.125182 4433 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.125200 4433 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.125216 4433 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.125236 4433 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.125247 4433 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.126220 4433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.126233 4433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.126974 4433 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.128690 4433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.128700 4433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.128796 4433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.128965 4433 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.128994 4433 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129086 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129098 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129105 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129110 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129116 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129121 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129127 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129132 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129139 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129145 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129154 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129171 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129203 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:04:51.129329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.129209 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:04:51.132490 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.132477 4433 server.go:1263] "Started kubelet" Oct 30 00:04:51.132801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.132531 4433 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:04:51.132801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.132612 4433 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:04:51.132801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.132683 4433 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:04:51.133001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.132971 4433 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:04:51.133106 ip-10-0-138-81 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:04:51.134038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.133631 4433 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:04:51.135097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.134833 4433 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:04:51.135097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.134840 4433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:04:51.135097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.134883 4433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:13 +0000 UTC" deadline="2028-06-10 17:17:51.064218883 +0000 UTC" Oct 30 00:04:51.135097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.134914 4433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14153h12m59.929308234s" Oct 30 00:04:51.135097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.134942 4433 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:04:51.135097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.134948 4433 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:04:51.135097 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:51.134981 4433 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-81.ec2.internal\" not found" Oct 30 00:04:51.136384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.135130 4433 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:04:51.137381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137339 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a79d2269-64f4-444e-9d57-c081a6744d81" volumeName="kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:04:51.137381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137379 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a79d2269-64f4-444e-9d57-c081a6744d81" volumeName="kubernetes.io/projected/a79d2269-64f4-444e-9d57-c081a6744d81-kube-api-access-xnvfv" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137395 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692f7a39-6f14-4a82-9990-60fd90fba46b" volumeName="kubernetes.io/projected/692f7a39-6f14-4a82-9990-60fd90fba46b-kube-api-access-zttcl" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137586 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce3b143-5c8f-4170-8f84-095f7668c53d" volumeName="kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-tuned" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137605 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f828c6e-3ce7-4457-b47a-fd34e484f0cd" volumeName="kubernetes.io/empty-dir/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-tmp-dir" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137641 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f828c6e-3ce7-4457-b47a-fd34e484f0cd" volumeName="kubernetes.io/projected/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-kube-api-access-4g4pl" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137656 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5996d3a1-c3de-4092-ba1d-71f041e7c37b" volumeName="kubernetes.io/empty-dir/5996d3a1-c3de-4092-ba1d-71f041e7c37b-data-volume" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137676 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aad4aae-3822-4a95-aa08-ecc8fcce7bee" volumeName="kubernetes.io/empty-dir/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-tmp-dir" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137692 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9046ee60-4e65-48c9-ab07-21171af6ce8d" volumeName="kubernetes.io/projected/9046ee60-4e65-48c9-ab07-21171af6ce8d-kube-api-access-nwl89" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137712 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="079ae719-a70f-4d46-923c-aeb73ce07125" volumeName="kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-config" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137727 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5996d3a1-c3de-4092-ba1d-71f041e7c37b" volumeName="kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137741 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce3b143-5c8f-4170-8f84-095f7668c53d" volumeName="kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-tmp" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137760 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce3b143-5c8f-4170-8f84-095f7668c53d" volumeName="kubernetes.io/projected/8ce3b143-5c8f-4170-8f84-095f7668c53d-kube-api-access-cxl8s" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137772 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9046ee60-4e65-48c9-ab07-21171af6ce8d" volumeName="kubernetes.io/configmap/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-cabundle" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137796 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26b4ad3c-8956-4e4a-a68f-2df514e7dc25" volumeName="kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137809 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3740a0a7-2025-4cfb-aa4f-9bcbfaf68780" volumeName="kubernetes.io/configmap/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-konnectivity-ca" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137827 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557dfb83-6842-4e18-b8e2-3426d7ba17ed" volumeName="kubernetes.io/projected/557dfb83-6842-4e18-b8e2-3426d7ba17ed-kube-api-access-kc54t" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137841 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a37bd471-b857-4907-b357-aae3315a2721" volumeName="kubernetes.io/projected/a37bd471-b857-4907-b357-aae3315a2721-kube-api-access-29b9r" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137860 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="079ae719-a70f-4d46-923c-aeb73ce07125" volumeName="kubernetes.io/projected/079ae719-a70f-4d46-923c-aeb73ce07125-kube-api-access-nww5d" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.137878 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97b31541-b41e-482e-bda2-afb18784d5d4" volumeName="kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138795 4433 factory.go:153] Registering CRI-O factory Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138815 4433 factory.go:223] Registration of the crio container factory successfully Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138867 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ba5774d-8b53-4430-be28-ba07235cde9f" volumeName="kubernetes.io/projected/9ba5774d-8b53-4430-be28-ba07235cde9f-kube-api-access-86jfk" seLinuxMountContext="" Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138891 4433 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138903 4433 factory.go:55] Registering systemd factory Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138912 4433 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138929 4433 factory.go:103] Registering Raw factory Oct 30 00:04:51.138958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138950 4433 manager.go:1199] Started watching for new ooms in manager Oct 30 00:04:51.140234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.139275 4433 manager.go:319] Starting recovery of all containers Oct 30 00:04:51.140726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.138900 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a79d2269-64f4-444e-9d57-c081a6744d81" volumeName="kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-binary-copy" seLinuxMountContext="" Oct 30 00:04:51.140804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140768 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d080e6f8-404d-4b58-9f16-c4825e28a3c0" volumeName="kubernetes.io/projected/d080e6f8-404d-4b58-9f16-c4825e28a3c0-kube-api-access-fqf26" seLinuxMountContext="" Oct 30 00:04:51.140804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140794 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="679a9efd-b724-4566-91ac-ba5000919e94" volumeName="kubernetes.io/projected/679a9efd-b724-4566-91ac-ba5000919e94-kube-api-access-bhmkp" seLinuxMountContext="" Oct 30 00:04:51.140894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140817 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692f7a39-6f14-4a82-9990-60fd90fba46b" volumeName="kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-daemon-config" seLinuxMountContext="" Oct 30 00:04:51.140894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140833 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f828c6e-3ce7-4457-b47a-fd34e484f0cd" volumeName="kubernetes.io/configmap/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-config-volume" seLinuxMountContext="" Oct 30 00:04:51.140894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140852 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9046ee60-4e65-48c9-ab07-21171af6ce8d" volumeName="kubernetes.io/secret/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-key" seLinuxMountContext="" Oct 30 00:04:51.140894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140868 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ba5774d-8b53-4430-be28-ba07235cde9f" volumeName="kubernetes.io/secret/9ba5774d-8b53-4430-be28-ba07235cde9f-cert" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140899 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="079ae719-a70f-4d46-923c-aeb73ce07125" volumeName="kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-env-overrides" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140913 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="079ae719-a70f-4d46-923c-aeb73ce07125" volumeName="kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140933 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557dfb83-6842-4e18-b8e2-3426d7ba17ed" volumeName="kubernetes.io/configmap/557dfb83-6842-4e18-b8e2-3426d7ba17ed-serviceca" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140946 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a37bd471-b857-4907-b357-aae3315a2721" volumeName="kubernetes.io/configmap/a37bd471-b857-4907-b357-aae3315a2721-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140960 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="079ae719-a70f-4d46-923c-aeb73ce07125" volumeName="kubernetes.io/secret/079ae719-a70f-4d46-923c-aeb73ce07125-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140979 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3740a0a7-2025-4cfb-aa4f-9bcbfaf68780" volumeName="kubernetes.io/secret/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-agent-certs" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.140994 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5996d3a1-c3de-4092-ba1d-71f041e7c37b" volumeName="kubernetes.io/projected/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-api-access-fgb6t" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.141012 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692f7a39-6f14-4a82-9990-60fd90fba46b" volumeName="kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-cni-binary-copy" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.141026 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aad4aae-3822-4a95-aa08-ecc8fcce7bee" volumeName="kubernetes.io/projected/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-kube-api-access-dwjgg" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.141041 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a79d2269-64f4-444e-9d57-c081a6744d81" volumeName="kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.141101 4433 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:04:51.141125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.141114 4433 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:04:51.152403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.152375 4433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:04:51.160194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.160173 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:04:51.160823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.160802 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:04:51.171032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.171009 4433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:04:51.171032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.171033 4433 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:04:51.171185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.171054 4433 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:04:51.171185 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:51.171093 4433 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:04:51.183954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.183925 4433 generic.go:356] "Generic (PLEG): container finished" podID="a2f00d42d4a021e56a936cb97199ccd7" containerID="1952a754f4745df156211864bc246a9456fadbc0f5714b216c0819a1be720a0d" exitCode=0 Oct 30 00:04:51.188195 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.188171 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:04:51.188637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.188615 4433 generic.go:356] "Generic (PLEG): container finished" podID="079ae719-a70f-4d46-923c-aeb73ce07125" containerID="f9800843e617cc1ef778668ccf9fffe5386bd5899f845c9907c16161200aae72" exitCode=1 Oct 30 00:04:51.190171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.190147 4433 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:04:51.191043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.191021 4433 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:51.193823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.193792 4433 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="5a10f6f5c34ddf4b413aa9b89504f04cc63344318da9a563c5a17376ec2aa981" exitCode=0 Oct 30 00:04:51.193823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.193819 4433 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="e0ad7954b88d0794e076893aba62038604cbacd0aa084b10396d7653ac5c9b2b" exitCode=0 Oct 30 00:04:51.193990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.193831 4433 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="c3b69913970b9056e8cbf34e9b8d7bc8820027b443cafe3bbafc6cc17290ff75" exitCode=0 Oct 30 00:04:51.193990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.193842 4433 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="7fe5c0f647ba4798b6f250e5b7fe6b7a2479028e05d52325fa7d6575df21f8e9" exitCode=0 Oct 30 00:04:51.193990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.193851 4433 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="559fee6eb134d509b60fcd28cb49532b2c5c2273a04b5c04c8332107e7f7a396" exitCode=0 Oct 30 00:04:51.193990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.193860 4433 generic.go:356] "Generic (PLEG): container finished" podID="a79d2269-64f4-444e-9d57-c081a6744d81" containerID="9c6a58edbc0426daa1823e84d042c4c8cef780c477bf89a10b74156421fe39f8" exitCode=0 Oct 30 00:04:51.203699 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.203679 4433 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:51.204357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.204332 4433 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:51.204464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.204439 4433 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:51.211642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.211626 4433 manager.go:324] Recovery completed Oct 30 00:04:51.231102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231079 4433 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:04:51.231102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231096 4433 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:04:51.231286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231117 4433 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:04:51.231286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231242 4433 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:04:51.231286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231254 4433 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:04:51.231286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231272 4433 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:04:51.231286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231277 4433 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:04:51.231286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231283 4433 policy_none.go:50] "Start" Oct 30 00:04:51.231533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231291 4433 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:04:51.231533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231301 4433 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:04:51.231533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231423 4433 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:04:51.231533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231430 4433 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:04:51.231533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.231435 4433 policy_none.go:44] "Start" Oct 30 00:04:51.235361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.235335 4433 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:04:51.238540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.238516 4433 manager.go:341] "Starting Device Plugin manager" Oct 30 00:04:51.239204 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:51.239185 4433 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:04:51.239204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239205 4433 server.go:87] "Starting device plugin registration server" Oct 30 00:04:51.239508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239497 4433 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:04:51.239547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239510 4433 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:04:51.239662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239635 4433 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:04:51.239774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239739 4433 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:04:51.239774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239774 4433 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:04:51.239867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239781 4433 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:04:51.239917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.239831 4433 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:04:51.239763035Z","UUID":"2df38777-6bff-4095-8bb9-a60fcedcc8fc","Handler":null,"Name":"","Endpoint":""} Oct 30 00:04:51.241480 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:51.241464 4433 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:04:51.241565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.241468 4433 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:04:51.241565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.241540 4433 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:04:51.271559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.271488 4433 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal"] Oct 30 00:04:51.271963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.271929 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" event={"ID":"a2f00d42d4a021e56a936cb97199ccd7","Type":"ContainerStarted","Data":"7c3f3760a958f90e4193b106d8651550dbd82bff566c744ff48ce6b52be611d4"} Oct 30 00:04:51.272061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.271964 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" event={"ID":"a2f00d42d4a021e56a936cb97199ccd7","Type":"ContainerDied","Data":"1952a754f4745df156211864bc246a9456fadbc0f5714b216c0819a1be720a0d"} Oct 30 00:04:51.272061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.271986 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" event={"ID":"a2f00d42d4a021e56a936cb97199ccd7","Type":"ContainerStarted","Data":"6b5220d748d307bf569f444c283299a7bdee3b2baf90548fd8fee15f08cddead"} Oct 30 00:04:51.272061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.272035 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" event={"ID":"0e8ee27d044b1e7c2240420e5926cb72","Type":"ContainerStarted","Data":"8b41f27cd456d9cb8a4b94dcd27a60e20d5596142e0b043b47751a044a1b101a"} Oct 30 00:04:51.272061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.272051 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" event={"ID":"0e8ee27d044b1e7c2240420e5926cb72","Type":"ContainerStarted","Data":"50905d7f808b54d5972e65f64c2c810d18201ce809a5daca06d9a1c8fb0be857"} Oct 30 00:04:51.272241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.272076 4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.272302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.272285 4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.293939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.293916 4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:51.294043 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:51.293956 4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-81.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.294495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.294479 4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:51.294529 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:51.294513 4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.340077 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.340048 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:51.341305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.341289 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:51.341382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.341316 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:51.341382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.341326 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-81.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:51.341382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.341357 4433 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.377281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.377258 4433 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.377380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.377322 4433 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.390988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.390965 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.391069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.390993 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.391069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.391008 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e8ee27d044b1e7c2240420e5926cb72-config\") pod \"kube-apiserver-proxy-ip-10-0-138-81.ec2.internal\" (UID: \"0e8ee27d044b1e7c2240420e5926cb72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.491707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.491673 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.491707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.491705 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.491865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.491729 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e8ee27d044b1e7c2240420e5926cb72-config\") pod \"kube-apiserver-proxy-ip-10-0-138-81.ec2.internal\" (UID: \"0e8ee27d044b1e7c2240420e5926cb72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.491865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.491777 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e8ee27d044b1e7c2240420e5926cb72-config\") pod \"kube-apiserver-proxy-ip-10-0-138-81.ec2.internal\" (UID: \"0e8ee27d044b1e7c2240420e5926cb72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.491865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.491783 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:51.491865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:51.491800 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2f00d42d4a021e56a936cb97199ccd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" (UID: \"a2f00d42d4a021e56a936cb97199ccd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:52.126047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.126020 4433 apiserver.go:52] "Watching apiserver" Oct 30 00:04:52.168503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.168476 4433 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:04:52.168888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.168871 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-81.ec2.internal","openshift-ingress-canary/ingress-canary-wrvj9","openshift-multus/multus-additional-cni-plugins-q7q8g","openshift-dns/node-resolver-8vxtm","openshift-dns/dns-default-n2bm4","openshift-multus/network-metrics-daemon-msd6l","openshift-ovn-kubernetes/ovnkube-node-wr7jg","openshift-insights/insights-runtime-extractor-2mrhd","kube-system/global-pull-secret-syncer-2c7g8","kube-system/konnectivity-agent-ghj4h","openshift-cluster-node-tuning-operator/tuned-j2gmp","openshift-image-registry/node-ca-q84nm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal","openshift-service-ca/service-ca-5fffc97bdf-x9kkv","openshift-multus/multus-xc5n4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs","openshift-network-diagnostics/network-check-target-q9h2g","openshift-network-operator/iptables-alerter-swbtw"] Oct 30 00:04:52.170528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.170510 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:52.174089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.174068 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.175455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.175423 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.191196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191176 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thx59\"" Oct 30 00:04:52.191196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191191 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k8n8m\"" Oct 30 00:04:52.191355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191181 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xzk4\"" Oct 30 00:04:52.191355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191217 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h92x8\"" Oct 30 00:04:52.191355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191176 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:04:52.191355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191252 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pxfs7\"" Oct 30 00:04:52.191355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191264 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fmnxz\"" Oct 30 00:04:52.191516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191374 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6hlg\"" Oct 30 00:04:52.191948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.191925 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.192036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192011 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:04:52.192090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192047 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.192090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192077 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:04:52.192175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192087 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:04:52.192175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192134 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.192258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192205 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.192312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192257 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:04:52.192312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192281 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.192312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192307 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.192463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192451 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:04:52.192507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192471 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.192507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192490 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:04:52.192586 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.192538 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.199488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.199471 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8lkn\"" Oct 30 00:04:52.199770 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.199754 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:04:52.200355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200337 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:04:52.200355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200354 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvd28\"" Oct 30 00:04:52.200503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200379 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:04:52.200503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200438 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kct9w\"" Oct 30 00:04:52.200503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200449 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 00:04:52.200731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200527 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:04:52.200731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200535 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntcgl\"" Oct 30 00:04:52.200731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200543 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqlwc\"" Oct 30 00:04:52.200731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.200590 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:04:52.203666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.203638 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7fsr\"" Oct 30 00:04:52.203750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.203671 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:04:52.203750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.203675 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m7wf2\"" Oct 30 00:04:52.203750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.203691 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vr82\"" Oct 30 00:04:52.203750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.203729 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-whzhw\"" Oct 30 00:04:52.203895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.203767 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:04:52.209517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.209503 4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:52.232971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.232953 4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:52.233024 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.232992 4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-81.ec2.internal" Oct 30 00:04:52.236939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.236926 4433 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:04:52.292121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292097 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.292121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292106 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.292355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292107 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.292355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292269 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.292355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292298 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:04:52.292355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292313 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292373 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292381 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292394 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292373 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292379 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292415 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292442 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292447 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292457 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 00:04:52.292519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292467 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:04:52.292802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292588 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:04:52.292802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292633 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.292802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292684 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.292802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292706 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:04:52.292802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.292737 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 00:04:52.295524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295505 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nww5d\" (UniqueName: \"kubernetes.io/projected/079ae719-a70f-4d46-923c-aeb73ce07125-kube-api-access-nww5d\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.295626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295531 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-dbus\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.295626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295548 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.295626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295563 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4pl\" (UniqueName: \"kubernetes.io/projected/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-kube-api-access-4g4pl\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.295626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295600 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:52.295788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295644 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.295788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295719 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-host\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.295788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295746 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.295910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295783 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttcl\" (UniqueName: \"kubernetes.io/projected/692f7a39-6f14-4a82-9990-60fd90fba46b-kube-api-access-zttcl\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.295910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295822 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/557dfb83-6842-4e18-b8e2-3426d7ba17ed-serviceca\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:52.295910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295860 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5996d3a1-c3de-4092-ba1d-71f041e7c37b-data-volume\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.295910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295877 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-log-socket\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.295910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295893 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-cni-binary-copy\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.296131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295919 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-etc-kubernetes\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.296131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295934 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-device-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.296131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295960 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5996d3a1-c3de-4092-ba1d-71f041e7c37b-data-volume\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.296131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.295978 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-hosts-file\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:52.296131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296100 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.296281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296144 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.296281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296174 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-bin\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.296584 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296561 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/557dfb83-6842-4e18-b8e2-3426d7ba17ed-serviceca\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:52.296642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296625 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86jfk\" (UniqueName: \"kubernetes.io/projected/9ba5774d-8b53-4430-be28-ba07235cde9f-kube-api-access-86jfk\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:52.296676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296654 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-k8s-cni-cncf-io\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.296676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296631 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.296676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296671 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-conf-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.296769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296689 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-tmp-dir\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:52.296769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296631 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-cni-binary-copy\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.296769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296721 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-var-lib-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.296769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296757 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-tmp-dir\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:52.296885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296759 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-ovn\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.296885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296794 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-modprobe-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.296885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296826 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-kubernetes\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.296885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296865 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-conf\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.297011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296891 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.297011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296933 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmkp\" (UniqueName: \"kubernetes.io/projected/679a9efd-b724-4566-91ac-ba5000919e94-kube-api-access-bhmkp\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.297011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296955 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296974 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-lib-modules\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.297011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.296996 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-tmp\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.297157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297060 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-config-volume\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.297157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297062 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-tmp\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.297157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297079 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-tmp-dir\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.297157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297145 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-tmp-dir\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.297290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297208 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-multus\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.297290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297237 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29b9r\" (UniqueName: \"kubernetes.io/projected/a37bd471-b857-4907-b357-aae3315a2721-kube-api-access-29b9r\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:52.297290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297251 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-config-volume\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.297290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297261 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvfv\" (UniqueName: \"kubernetes.io/projected/a79d2269-64f4-444e-9d57-c081a6744d81-kube-api-access-xnvfv\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.297409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297342 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-systemd-units\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297442 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297428 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-env-overrides\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297474 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297457 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-script-lib\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297490 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-kubelet-config\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.297623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297606 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysconfig\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.297659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297632 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-env-overrides\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297639 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:52.297718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297693 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-konnectivity-ca\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:52.297763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297724 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-system-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.297763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297739 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-slash\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297775 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-script-lib\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297840 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-netns\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.297892 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297879 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.297930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297908 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-konnectivity-ca\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:52.297930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297921 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-run\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.297994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297944 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxl8s\" (UniqueName: \"kubernetes.io/projected/8ce3b143-5c8f-4170-8f84-095f7668c53d-kube-api-access-cxl8s\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.297994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297959 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-key\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:52.297994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.297974 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwl89\" (UniqueName: \"kubernetes.io/projected/9046ee60-4e65-48c9-ab07-21171af6ce8d-kube-api-access-nwl89\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:52.298112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298095 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97b31541-b41e-482e-bda2-afb18784d5d4-original-pull-secret\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.298112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298102 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-multus-certs\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.298168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298134 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37bd471-b857-4907-b357-aae3315a2721-iptables-alerter-script\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:52.298196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298166 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-system-cni-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.298235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298193 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5996d3a1-c3de-4092-ba1d-71f041e7c37b-crio-socket\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.298235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298195 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-key\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:52.298295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298245 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-etc-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.298295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298262 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-node-log\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.298295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298286 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-agent-certs\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:52.298377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298306 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37bd471-b857-4907-b357-aae3315a2721-host-slash\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:52.298377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298328 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.298377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298351 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-registration-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.298460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298381 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37bd471-b857-4907-b357-aae3315a2721-iptables-alerter-script\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:52.298460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298389 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjgg\" (UniqueName: \"kubernetes.io/projected/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-kube-api-access-dwjgg\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:52.298460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298438 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-netd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.298543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298512 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-var-lib-kubelet\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.298543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298511 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3740a0a7-2025-4cfb-aa4f-9bcbfaf68780-agent-certs\") pod \"konnectivity-agent-ghj4h\" (UID: \"3740a0a7-2025-4cfb-aa4f-9bcbfaf68780\") " pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:52.298543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298539 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.298638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298584 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/557dfb83-6842-4e18-b8e2-3426d7ba17ed-host\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:52.298638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298622 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-systemd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.298700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298644 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-tuned\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.298700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298675 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ba5774d-8b53-4430-be28-ba07235cde9f-cert\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:52.298752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298711 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.298752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298713 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-tuned\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.298752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298720 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-sys-fs\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.298839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298820 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.298888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298871 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqf26\" (UniqueName: \"kubernetes.io/projected/d080e6f8-404d-4b58-9f16-c4825e28a3c0-kube-api-access-fqf26\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:52.298888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298882 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ba5774d-8b53-4430-be28-ba07235cde9f-cert\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:52.298948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298910 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:52.299009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.298995 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-socket-dir-parent\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299038 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-daemon-config\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299069 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-cnibin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299098 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-bin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299140 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-cnibin\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.299182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299140 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9046ee60-4e65-48c9-ab07-21171af6ce8d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:52.299182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299171 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-daemon-config\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299178 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.299235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299209 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgb6t\" (UniqueName: \"kubernetes.io/projected/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-api-access-fgb6t\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.299235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299227 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079ae719-a70f-4d46-923c-aeb73ce07125-ovn-node-metrics-cert\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.299315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299242 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-systemd\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.299315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299263 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-sys\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.299367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299309 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-netns\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299327 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.299367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299342 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-kubelet\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299373 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079ae719-a70f-4d46-923c-aeb73ce07125-ovn-node-metrics-cert\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.299445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299383 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-hostroot\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.299445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299412 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.299445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299430 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-socket-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.299557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299459 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.299557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299548 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-kubelet\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.299632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299592 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a79d2269-64f4-444e-9d57-c081a6744d81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.299632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299600 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.299690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299644 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc54t\" (UniqueName: \"kubernetes.io/projected/557dfb83-6842-4e18-b8e2-3426d7ba17ed-kube-api-access-kc54t\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:52.299690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299669 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-config\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.299819 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.299805 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079ae719-a70f-4d46-923c-aeb73ce07125-ovnkube-config\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.317424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.317395 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4pl\" (UniqueName: \"kubernetes.io/projected/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-kube-api-access-4g4pl\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.317508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.317467 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttcl\" (UniqueName: \"kubernetes.io/projected/692f7a39-6f14-4a82-9990-60fd90fba46b-kube-api-access-zttcl\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.320590 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.318288 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872tn\" (UniqueName: \"kubernetes.io/projected/26b4ad3c-8956-4e4a-a68f-2df514e7dc25-kube-api-access-872tn\") pod \"network-check-target-q9h2g\" (UID: \"26b4ad3c-8956-4e4a-a68f-2df514e7dc25\") " pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:52.320590 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.318507 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nww5d\" (UniqueName: \"kubernetes.io/projected/079ae719-a70f-4d46-923c-aeb73ce07125-kube-api-access-nww5d\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.321687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.321661 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqf26\" (UniqueName: \"kubernetes.io/projected/d080e6f8-404d-4b58-9f16-c4825e28a3c0-kube-api-access-fqf26\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:52.321805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.321737 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmkp\" (UniqueName: \"kubernetes.io/projected/679a9efd-b724-4566-91ac-ba5000919e94-kube-api-access-bhmkp\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.321805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.321759 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjgg\" (UniqueName: \"kubernetes.io/projected/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-kube-api-access-dwjgg\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:52.321989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.321971 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86jfk\" (UniqueName: \"kubernetes.io/projected/9ba5774d-8b53-4430-be28-ba07235cde9f-kube-api-access-86jfk\") pod \"ingress-canary-wrvj9\" (UID: \"9ba5774d-8b53-4430-be28-ba07235cde9f\") " pod="openshift-ingress-canary/ingress-canary-wrvj9" Oct 30 00:04:52.322148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.322130 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwl89\" (UniqueName: \"kubernetes.io/projected/9046ee60-4e65-48c9-ab07-21171af6ce8d-kube-api-access-nwl89\") pod \"service-ca-5fffc97bdf-x9kkv\" (UID: \"9046ee60-4e65-48c9-ab07-21171af6ce8d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x9kkv" Oct 30 00:04:52.322506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.322487 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29b9r\" (UniqueName: \"kubernetes.io/projected/a37bd471-b857-4907-b357-aae3315a2721-kube-api-access-29b9r\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:52.322605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.322562 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxl8s\" (UniqueName: \"kubernetes.io/projected/8ce3b143-5c8f-4170-8f84-095f7668c53d-kube-api-access-cxl8s\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.322729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.322711 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvfv\" (UniqueName: \"kubernetes.io/projected/a79d2269-64f4-444e-9d57-c081a6744d81-kube-api-access-xnvfv\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.323418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.323399 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgb6t\" (UniqueName: \"kubernetes.io/projected/5996d3a1-c3de-4092-ba1d-71f041e7c37b-kube-api-access-fgb6t\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.324036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.324020 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc54t\" (UniqueName: \"kubernetes.io/projected/557dfb83-6842-4e18-b8e2-3426d7ba17ed-kube-api-access-kc54t\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:52.400444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400364 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-socket-dir-parent\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400393 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-cnibin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400412 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-bin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400429 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-cnibin\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400449 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-systemd\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400469 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-sys\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400491 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-netns\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400486 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-bin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400514 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-kubelet\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400507 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-cnibin\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400505 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-cnibin\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400548 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-hostroot\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400551 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-kubelet\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400554 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-socket-dir-parent\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400557 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-sys\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400562 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-netns\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400609 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-socket-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400619 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-hostroot\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400625 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400683 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-systemd\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400696 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-kubelet\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400688 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400706 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-socket-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400716 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400743 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400742 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-kubelet\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400761 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-dbus\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.400781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400789 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400813 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400835 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-host\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400857 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400860 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-dbus\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400881 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-log-socket\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400900 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400908 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-host\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400904 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-etc-kubernetes\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400938 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-device-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.400954 4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400961 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400969 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-device-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.400997 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-etc-kubernetes\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401015 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-log-socket\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401020 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-hosts-file\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.401057 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls podName:8f828c6e-3ce7-4457-b47a-fd34e484f0cd nodeName:}" failed. No retries permitted until 2026-10-30 00:04:52.901034786 +0000 UTC m=+1.835103312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls") pod "dns-default-n2bm4" (UID: "8f828c6e-3ce7-4457-b47a-fd34e484f0cd") : secret "dns-default-metrics-tls" not found Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401064 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aad4aae-3822-4a95-aa08-ecc8fcce7bee-hosts-file\") pod \"node-resolver-8vxtm\" (UID: \"8aad4aae-3822-4a95-aa08-ecc8fcce7bee\") " pod="openshift-dns/node-resolver-8vxtm" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401095 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401122 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-bin\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401147 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-k8s-cni-cncf-io\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401171 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-conf-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401189 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-bin\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.401196 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401203 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-k8s-cni-cncf-io\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401223 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-multus-conf-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401235 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-var-lib-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401196 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-var-lib-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.401254 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:52.901237701 +0000 UTC m=+1.835306220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401288 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-ovn\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401312 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-modprobe-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401320 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-ovn\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401334 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-kubernetes\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401356 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-conf\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401378 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401403 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401410 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-kubernetes\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401426 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-lib-modules\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401445 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401449 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401453 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-multus\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401485 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-modprobe-d\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401494 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-var-lib-cni-multus\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401499 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysctl-conf\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401495 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-systemd-units\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401521 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-systemd-units\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401524 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-lib-modules\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401545 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-kubelet-config\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401591 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysconfig\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401611 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97b31541-b41e-482e-bda2-afb18784d5d4-kubelet-config\") pod \"global-pull-secret-syncer-2c7g8\" (UID: \"97b31541-b41e-482e-bda2-afb18784d5d4\") " pod="kube-system/global-pull-secret-syncer-2c7g8" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401626 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401652 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-system-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401677 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-slash\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401683 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-etc-sysconfig\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401700 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-netns\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401723 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-run\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401728 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-system-cni-dir\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.401740 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401750 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-multus-certs\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401765 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-netns\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401773 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-system-cni-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.401790 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:52.901775 +0000 UTC m=+1.835843518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : secret "metrics-daemon-secret" not found Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401805 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/692f7a39-6f14-4a82-9990-60fd90fba46b-host-run-multus-certs\") pod \"multus-xc5n4\" (UID: \"692f7a39-6f14-4a82-9990-60fd90fba46b\") " pod="openshift-multus/multus-xc5n4" Oct 30 00:04:52.401759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401808 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-system-cni-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401810 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-run\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401828 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5996d3a1-c3de-4092-ba1d-71f041e7c37b-crio-socket\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401742 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-slash\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401846 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-etc-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401861 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-node-log\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401883 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37bd471-b857-4907-b357-aae3315a2721-host-slash\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401907 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401912 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-etc-openvswitch\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401914 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-node-log\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401937 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-registration-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401955 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37bd471-b857-4907-b357-aae3315a2721-host-slash\") pod \"iptables-alerter-swbtw\" (UID: \"a37bd471-b857-4907-b357-aae3315a2721\") " pod="openshift-network-operator/iptables-alerter-swbtw" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401963 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-netd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401978 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5996d3a1-c3de-4092-ba1d-71f041e7c37b-crio-socket\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.401999 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-var-lib-kubelet\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402004 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a79d2269-64f4-444e-9d57-c081a6744d81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7q8g\" (UID: \"a79d2269-64f4-444e-9d57-c081a6744d81\") " pod="openshift-multus/multus-additional-cni-plugins-q7q8g" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402003 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-cni-netd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402025 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/557dfb83-6842-4e18-b8e2-3426d7ba17ed-host\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402032 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-registration-dir\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402048 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-systemd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402082 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-run-systemd\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402089 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3b143-5c8f-4170-8f84-095f7668c53d-var-lib-kubelet\") pod \"tuned-j2gmp\" (UID: \"8ce3b143-5c8f-4170-8f84-095f7668c53d\") " pod="openshift-cluster-node-tuning-operator/tuned-j2gmp" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402097 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/557dfb83-6842-4e18-b8e2-3426d7ba17ed-host\") pod \"node-ca-q84nm\" (UID: \"557dfb83-6842-4e18-b8e2-3426d7ba17ed\") " pod="openshift-image-registry/node-ca-q84nm" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402118 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-sys-fs\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402144 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402203 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079ae719-a70f-4d46-923c-aeb73ce07125-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr7jg\" (UID: \"079ae719-a70f-4d46-923c-aeb73ce07125\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:52.403648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.402208 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/679a9efd-b724-4566-91ac-ba5000919e94-sys-fs\") pod \"aws-ebs-csi-driver-node-cggjs\" (UID: \"679a9efd-b724-4566-91ac-ba5000919e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cggjs" Oct 30 00:04:52.464407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.464378 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:52.904714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.904683 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.904913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.904727 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:52.904913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.904763 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:52.904913 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.904869 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:52.904913 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.904879 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:52.905063 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.904934 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.904920228 +0000 UTC m=+2.838988746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:52.905063 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:52.904949 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.904943266 +0000 UTC m=+2.839011779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : secret "metrics-daemon-secret" not found Oct 30 00:04:52.905135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.905073 4433 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:04:52.907107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.907078 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f828c6e-3ce7-4457-b47a-fd34e484f0cd-metrics-tls\") pod \"dns-default-n2bm4\" (UID: \"8f828c6e-3ce7-4457-b47a-fd34e484f0cd\") " pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:52.927654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:52.927628 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:53.050887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.050845 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:53.076880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.076859 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:53.202506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.202482 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:53.253121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.253078 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:53.402833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.402801 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2bm4"] Oct 30 00:04:53.406353 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:53.406284 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f828c6e_3ce7_4457_b47a_fd34e484f0cd.slice/crio-ab6460060ce5c62290274baa2afaf7f8830aec120c6820b39440b0ffe6c61385 WatchSource:0}: Error finding container ab6460060ce5c62290274baa2afaf7f8830aec120c6820b39440b0ffe6c61385: Status 404 returned error can't find the container with id ab6460060ce5c62290274baa2afaf7f8830aec120c6820b39440b0ffe6c61385 Oct 30 00:04:53.408631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.408612 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:04:53.655762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.655722 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:53.854493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.854446 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:53.912316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.912290 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:53.912416 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:53.912328 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:53.912462 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:53.912432 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:53.912462 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:53.912437 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:53.912517 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:53.912483 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs podName:d080e6f8-404d-4b58-9f16-c4825e28a3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:55.912470277 +0000 UTC m=+4.846538790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs") pod "network-metrics-daemon-msd6l" (UID: "d080e6f8-404d-4b58-9f16-c4825e28a3c0") : secret "metrics-daemon-secret" not found Oct 30 00:04:53.912517 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:04:53.912496 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls podName:5996d3a1-c3de-4092-ba1d-71f041e7c37b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:55.912489624 +0000 UTC m=+4.846558137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mrhd" (UID: "5996d3a1-c3de-4092-ba1d-71f041e7c37b") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:54.215285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:54.215200 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2bm4" event={"ID":"8f828c6e-3ce7-4457-b47a-fd34e484f0cd","Type":"ContainerStarted","Data":"ab6460060ce5c62290274baa2afaf7f8830aec120c6820b39440b0ffe6c61385"} Oct 30 00:04:54.451721 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:54.451689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:54.652712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:54.652684 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:54.652922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:54.652910 4433 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:55.056742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.056659 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:55.224358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.224325 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2bm4" event={"ID":"8f828c6e-3ce7-4457-b47a-fd34e484f0cd","Type":"ContainerStarted","Data":"fa2524fce57f96b72e7a45f0899a51877e1d13ba61b320fe7415e996beaa0e37"} Oct 30 00:04:55.252335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.252302 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:55.355285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.355256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:55.480410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.480372 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:55.855020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.854990 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:55.926933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.926900 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:55.927139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.926952 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:55.929303 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.929265 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d080e6f8-404d-4b58-9f16-c4825e28a3c0-metrics-certs\") pod \"network-metrics-daemon-msd6l\" (UID: \"d080e6f8-404d-4b58-9f16-c4825e28a3c0\") " pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:55.929409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:55.929348 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5996d3a1-c3de-4092-ba1d-71f041e7c37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mrhd\" (UID: \"5996d3a1-c3de-4092-ba1d-71f041e7c37b\") " pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:56.071797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.071757 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msd6l" Oct 30 00:04:56.074825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.074804 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2mrhd" Oct 30 00:04:56.209524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.209489 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:04:56.228862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.228835 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2bm4" event={"ID":"8f828c6e-3ce7-4457-b47a-fd34e484f0cd","Type":"ContainerStarted","Data":"bbce00921fbdbab1daa0a9f26594bca4af387da43222c041446f0dce0af7fe2c"} Oct 30 00:04:56.229223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.228957 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n2bm4" Oct 30 00:04:56.379003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.378931 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-msd6l"] Oct 30 00:04:56.381949 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:56.381913 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd080e6f8_404d_4b58_9f16_c4825e28a3c0.slice/crio-ea5f5be98b8045a656e8d4b6a1d515d75af4ab60fa0d35a0d1bcad85a612e526 WatchSource:0}: Error finding container ea5f5be98b8045a656e8d4b6a1d515d75af4ab60fa0d35a0d1bcad85a612e526: Status 404 returned error can't find the container with id ea5f5be98b8045a656e8d4b6a1d515d75af4ab60fa0d35a0d1bcad85a612e526 Oct 30 00:04:56.384962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.384921 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2mrhd"] Oct 30 00:04:56.389048 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:04:56.389015 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5996d3a1_c3de_4092_ba1d_71f041e7c37b.slice/crio-dfc82c9bd95453963e4c966935a572fecf5f7bd3dbb5b1defd0f9b877dfb8bde WatchSource:0}: Error finding container dfc82c9bd95453963e4c966935a572fecf5f7bd3dbb5b1defd0f9b877dfb8bde: Status 404 returned error can't find the container with id dfc82c9bd95453963e4c966935a572fecf5f7bd3dbb5b1defd0f9b877dfb8bde Oct 30 00:04:56.559716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.559678 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:56.644454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.644382 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ghj4h" Oct 30 00:04:56.666133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.666100 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:56.859672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.859628 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:56.978341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:56.978227 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:57.086225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.086171 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:57.181007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.180982 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr7jg" Oct 30 00:04:57.232894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.232822 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-msd6l" event={"ID":"d080e6f8-404d-4b58-9f16-c4825e28a3c0","Type":"ContainerStarted","Data":"ea5f5be98b8045a656e8d4b6a1d515d75af4ab60fa0d35a0d1bcad85a612e526"} Oct 30 00:04:57.234560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.234537 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mrhd" event={"ID":"5996d3a1-c3de-4092-ba1d-71f041e7c37b","Type":"ContainerStarted","Data":"a7ee7ed12fd233abe2f5c821df168a1ee7d9d5e81239dbe12095130e48ac87bc"} Oct 30 00:04:57.234782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.234598 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mrhd" event={"ID":"5996d3a1-c3de-4092-ba1d-71f041e7c37b","Type":"ContainerStarted","Data":"b2df578d125add6a0a1c2b9b955f0367b7afa4f60a4f64240d107dadadb62cba"} Oct 30 00:04:57.234782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.234618 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mrhd" event={"ID":"5996d3a1-c3de-4092-ba1d-71f041e7c37b","Type":"ContainerStarted","Data":"dfc82c9bd95453963e4c966935a572fecf5f7bd3dbb5b1defd0f9b877dfb8bde"} Oct 30 00:04:57.257670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.257642 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:57.662698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.662660 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:57.770658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.770625 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:57.871255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.871222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:57.975106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:57.975016 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:58.079781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:58.079737 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:58.206200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:58.206164 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:58.244078 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:58.244003 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-msd6l" event={"ID":"d080e6f8-404d-4b58-9f16-c4825e28a3c0","Type":"ContainerStarted","Data":"7ebd23979d46ea05d0dbaed3b8e257c371aac3e19c1d951ea3ebc984a05f9aba"} Oct 30 00:04:58.244509 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:58.244093 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-msd6l" event={"ID":"d080e6f8-404d-4b58-9f16-c4825e28a3c0","Type":"ContainerStarted","Data":"ff221ca2bfee1f0fc611fb67d11a416ff8f65bc348a4066bfb06d20b22367d3f"} Oct 30 00:04:58.411158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:58.411124 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:58.556419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:58.556343 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:59.249244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:04:59.249174 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mrhd" event={"ID":"5996d3a1-c3de-4092-ba1d-71f041e7c37b","Type":"ContainerStarted","Data":"40f7028e38947b5717a2aab7f3eda2e117f1daafd455998d119da0edad8ffa60"} Oct 30 00:05:02.837313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.837278 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m"] Oct 30 00:05:02.839223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.839206 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" Oct 30 00:05:02.842450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.842427 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-hjwsx"] Oct 30 00:05:02.844019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.844005 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:02.870691 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.870667 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vfpm8\"" Oct 30 00:05:02.870793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.870691 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 00:05:02.903381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.903358 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qx55q\"" Oct 30 00:05:02.903860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.903842 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 00:05:02.903860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.903842 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 00:05:02.904163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.904145 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m"] Oct 30 00:05:02.917226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.917203 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hjwsx"] Oct 30 00:05:02.973191 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.973163 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnqj\" (UniqueName: \"kubernetes.io/projected/0ef650f7-6cd3-4df0-8be4-5bc699dac77e-kube-api-access-zfnqj\") pod \"downloads-6dd789c8b7-hjwsx\" (UID: \"0ef650f7-6cd3-4df0-8be4-5bc699dac77e\") " pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:02.973313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.973198 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e8b990f-be4c-434b-8966-5ebdb335e811-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mv64m\" (UID: \"2e8b990f-be4c-434b-8966-5ebdb335e811\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" Oct 30 00:05:02.973313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:02.973237 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ef650f7-6cd3-4df0-8be4-5bc699dac77e-tmp\") pod \"downloads-6dd789c8b7-hjwsx\" (UID: \"0ef650f7-6cd3-4df0-8be4-5bc699dac77e\") " pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:03.010318 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.010294 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:03.074335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.074305 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ef650f7-6cd3-4df0-8be4-5bc699dac77e-tmp\") pod \"downloads-6dd789c8b7-hjwsx\" (UID: \"0ef650f7-6cd3-4df0-8be4-5bc699dac77e\") " pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:03.074482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.074340 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnqj\" (UniqueName: \"kubernetes.io/projected/0ef650f7-6cd3-4df0-8be4-5bc699dac77e-kube-api-access-zfnqj\") pod \"downloads-6dd789c8b7-hjwsx\" (UID: \"0ef650f7-6cd3-4df0-8be4-5bc699dac77e\") " pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:03.074482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.074366 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e8b990f-be4c-434b-8966-5ebdb335e811-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mv64m\" (UID: \"2e8b990f-be4c-434b-8966-5ebdb335e811\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" Oct 30 00:05:03.074744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.074727 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ef650f7-6cd3-4df0-8be4-5bc699dac77e-tmp\") pod \"downloads-6dd789c8b7-hjwsx\" (UID: \"0ef650f7-6cd3-4df0-8be4-5bc699dac77e\") " pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:03.076651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.076630 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e8b990f-be4c-434b-8966-5ebdb335e811-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mv64m\" (UID: \"2e8b990f-be4c-434b-8966-5ebdb335e811\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" Oct 30 00:05:03.101115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.101055 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnqj\" (UniqueName: \"kubernetes.io/projected/0ef650f7-6cd3-4df0-8be4-5bc699dac77e-kube-api-access-zfnqj\") pod \"downloads-6dd789c8b7-hjwsx\" (UID: \"0ef650f7-6cd3-4df0-8be4-5bc699dac77e\") " pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:03.119287 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.119264 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:03.150259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.150230 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" Oct 30 00:05:03.155346 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.155319 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:03.234531 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.234505 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:03.455037 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.454974 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m"] Oct 30 00:05:03.457868 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:03.457837 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8b990f_be4c_434b_8966_5ebdb335e811.slice/crio-1ad5a76d1f522302a4c9e4dbbb6796b1fe6a974372ee9cca942cbc0c703c8ffc WatchSource:0}: Error finding container 1ad5a76d1f522302a4c9e4dbbb6796b1fe6a974372ee9cca942cbc0c703c8ffc: Status 404 returned error can't find the container with id 1ad5a76d1f522302a4c9e4dbbb6796b1fe6a974372ee9cca942cbc0c703c8ffc Oct 30 00:05:03.461097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:03.461076 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hjwsx"] Oct 30 00:05:03.463671 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:03.463641 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef650f7_6cd3_4df0_8be4_5bc699dac77e.slice/crio-138156cc89ee94e1e3ab7ee10f9d79873331be7617a090246862bafa4bcc0c97 WatchSource:0}: Error finding container 138156cc89ee94e1e3ab7ee10f9d79873331be7617a090246862bafa4bcc0c97: Status 404 returned error can't find the container with id 138156cc89ee94e1e3ab7ee10f9d79873331be7617a090246862bafa4bcc0c97 Oct 30 00:05:04.263814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:04.263745 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" event={"ID":"2e8b990f-be4c-434b-8966-5ebdb335e811","Type":"ContainerStarted","Data":"1ad5a76d1f522302a4c9e4dbbb6796b1fe6a974372ee9cca942cbc0c703c8ffc"} Oct 30 00:05:04.265151 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:04.265107 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hjwsx" event={"ID":"0ef650f7-6cd3-4df0-8be4-5bc699dac77e","Type":"ContainerStarted","Data":"138156cc89ee94e1e3ab7ee10f9d79873331be7617a090246862bafa4bcc0c97"} Oct 30 00:05:05.269486 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:05.269448 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" event={"ID":"2e8b990f-be4c-434b-8966-5ebdb335e811","Type":"ContainerStarted","Data":"5b93b83359fb5d3f7bd06c6a5afb2cb32dc7dd5dcde3ee9745c20c16dcacee7c"} Oct 30 00:05:05.269947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:05.269716 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" Oct 30 00:05:05.275406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:05.275380 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" Oct 30 00:05:05.315688 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:05.315636 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mv64m" podStartSLOduration=2.265212498 podStartE2EDuration="3.315621081s" totalImagesPullingTime="1.050408583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:05.315024139 +0000 UTC m=+14.249092675" watchObservedRunningTime="2026-10-30 00:05:05.315621081 +0000 UTC m=+14.249689619" Oct 30 00:05:05.447344 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:05.447301 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:05.581334 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:05.581242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:06.108460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.108357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:06.208967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.208937 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr"] Oct 30 00:05:06.211168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.211144 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.215210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.215181 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:06.236363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.236342 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n2bm4" Oct 30 00:05:06.243779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.243436 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:05:06.243779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.243455 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:05:06.243779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.243473 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:05:06.243779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.243516 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 00:05:06.243779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.243535 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 00:05:06.243779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.243549 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pfrj5\"" Oct 30 00:05:06.244152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.243955 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr"] Oct 30 00:05:06.295715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.295680 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cab57612-8d11-4773-a8c6-18c4fa22458a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.296166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.295737 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cab57612-8d11-4773-a8c6-18c4fa22458a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.296166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.295764 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cab57612-8d11-4773-a8c6-18c4fa22458a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.296166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.295889 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgrk\" (UniqueName: \"kubernetes.io/projected/cab57612-8d11-4773-a8c6-18c4fa22458a-kube-api-access-qjgrk\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.341249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.341214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:06.396385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.396297 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cab57612-8d11-4773-a8c6-18c4fa22458a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.396385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.396338 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cab57612-8d11-4773-a8c6-18c4fa22458a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.396565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.396367 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cab57612-8d11-4773-a8c6-18c4fa22458a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.396684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.396651 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgrk\" (UniqueName: \"kubernetes.io/projected/cab57612-8d11-4773-a8c6-18c4fa22458a-kube-api-access-qjgrk\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.397124 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.397102 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cab57612-8d11-4773-a8c6-18c4fa22458a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.399161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.399137 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cab57612-8d11-4773-a8c6-18c4fa22458a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.399274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.399145 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cab57612-8d11-4773-a8c6-18c4fa22458a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.425237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.425209 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgrk\" (UniqueName: \"kubernetes.io/projected/cab57612-8d11-4773-a8c6-18c4fa22458a-kube-api-access-qjgrk\") pod \"prometheus-operator-9cd89b74c-4r6fr\" (UID: \"cab57612-8d11-4773-a8c6-18c4fa22458a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.525225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.525186 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" Oct 30 00:05:06.826393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:06.826358 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr"] Oct 30 00:05:06.830357 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:06.830326 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab57612_8d11_4773_a8c6_18c4fa22458a.slice/crio-30673cfc54366f5d65517044c314a4df496e77b74fe5e6ec10f4e9f4adbbcedd WatchSource:0}: Error finding container 30673cfc54366f5d65517044c314a4df496e77b74fe5e6ec10f4e9f4adbbcedd: Status 404 returned error can't find the container with id 30673cfc54366f5d65517044c314a4df496e77b74fe5e6ec10f4e9f4adbbcedd Oct 30 00:05:07.100589 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:05:07.100496 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:05:07.277546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:07.277513 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" event={"ID":"cab57612-8d11-4773-a8c6-18c4fa22458a","Type":"ContainerStarted","Data":"30673cfc54366f5d65517044c314a4df496e77b74fe5e6ec10f4e9f4adbbcedd"} Oct 30 00:05:08.283768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:08.283740 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" event={"ID":"cab57612-8d11-4773-a8c6-18c4fa22458a","Type":"ContainerStarted","Data":"4dc67f0dee1b9130bbf4fa1f3d99aeb8d80b304b972d150e1951d174cd9bb39f"} Oct 30 00:05:08.284082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:08.283790 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" event={"ID":"cab57612-8d11-4773-a8c6-18c4fa22458a","Type":"ContainerStarted","Data":"5e524af4f771ad4d3c26b3162bb89aadf62a5e2abfdb4e42320d70616704949a"} Oct 30 00:05:08.349794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:08.349746 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4r6fr" podStartSLOduration=1.026746856 podStartE2EDuration="2.349731246s" totalImagesPullingTime="1.32298439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:08.348869138 +0000 UTC m=+17.282937694" watchObservedRunningTime="2026-10-30 00:05:08.349731246 +0000 UTC m=+17.283799772" Oct 30 00:05:09.093374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:09.093331 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:10.615637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.615594 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:10.732070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.732020 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:10.871929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.871845 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:10.938626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.937732 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp"] Oct 30 00:05:10.940525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.939956 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:10.967191 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.967166 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp"] Oct 30 00:05:10.978322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.978207 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bw9xv\"" Oct 30 00:05:10.978322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.978285 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:05:10.978322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:10.978292 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 00:05:11.010361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.010327 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rc8jd"] Oct 30 00:05:11.013041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.013012 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.015614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.015554 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:11.033105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.033079 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.033200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.033113 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.033245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.033205 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5gr\" (UniqueName: \"kubernetes.io/projected/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-kube-api-access-rd5gr\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.033245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.033227 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.048071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.048047 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:05:11.065471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.065447 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:05:11.065471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.065461 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:05:11.065749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.065474 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x8k7p\"" Oct 30 00:05:11.133838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133771 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.133838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133813 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqzh\" (UniqueName: \"kubernetes.io/projected/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-kube-api-access-2nqzh\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133843 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-sys\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133862 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133885 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5gr\" (UniqueName: \"kubernetes.io/projected/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-kube-api-access-rd5gr\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133907 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-tls\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133931 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133962 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133977 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-metrics-client-ca\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.133999 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-wtmp\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.134016 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.134051 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-textfile\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.134085 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-root\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.134554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.134532 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.136928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.136901 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:11.137068 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.137043 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.137455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.137437 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.158382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.158358 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5gr\" (UniqueName: \"kubernetes.io/projected/f3e8bfb3-fa1f-4af9-81ec-32c946bc7919-kube-api-access-rd5gr\") pod \"openshift-state-metrics-7bd7489957-vt9fp\" (UID: \"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.234542 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234516 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqzh\" (UniqueName: \"kubernetes.io/projected/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-kube-api-access-2nqzh\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234554 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-sys\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234594 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234620 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-tls\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234641 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-metrics-client-ca\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234659 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-wtmp\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234680 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234700 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-textfile\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234719 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-root\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234786 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-root\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.234955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.234823 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-sys\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.235436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.235412 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-textfile\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.235545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.235414 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-metrics-client-ca\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.235636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.235611 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-wtmp\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.235938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.235891 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.237751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.237720 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.238626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.238585 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-node-exporter-tls\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.255806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.255787 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" Oct 30 00:05:11.265044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.259364 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqzh\" (UniqueName: \"kubernetes.io/projected/3d5c2cbc-2adb-4ad2-bb48-87071c4497db-kube-api-access-2nqzh\") pod \"node-exporter-rc8jd\" (UID: \"3d5c2cbc-2adb-4ad2-bb48-87071c4497db\") " pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.301543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.301406 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:11.330776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.329426 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rc8jd" Oct 30 00:05:11.343640 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:11.343433 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5c2cbc_2adb_4ad2_bb48_87071c4497db.slice/crio-4b6d3a5f29f6eedb992f07680611a288e803edd86a8d8d4ca6303d743f8a4c1b WatchSource:0}: Error finding container 4b6d3a5f29f6eedb992f07680611a288e803edd86a8d8d4ca6303d743f8a4c1b: Status 404 returned error can't find the container with id 4b6d3a5f29f6eedb992f07680611a288e803edd86a8d8d4ca6303d743f8a4c1b Oct 30 00:05:11.468799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.468721 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:11.577275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.577244 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp"] Oct 30 00:05:11.580356 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:11.580331 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e8bfb3_fa1f_4af9_81ec_32c946bc7919.slice/crio-530eb448ac7e4084e6c104fc3231a5cc9a372ce97223d50eb8f7dc20d7843c98 WatchSource:0}: Error finding container 530eb448ac7e4084e6c104fc3231a5cc9a372ce97223d50eb8f7dc20d7843c98: Status 404 returned error can't find the container with id 530eb448ac7e4084e6c104fc3231a5cc9a372ce97223d50eb8f7dc20d7843c98 Oct 30 00:05:11.975067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:11.975004 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:12.227582 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.227371 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:12.294655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.294621 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:05:12.297686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.297660 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.305477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.305447 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" event={"ID":"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919","Type":"ContainerStarted","Data":"4892a4e01bf4fce8895eaaa65523bb857a899686ae3cc67fab32b6eb646453b2"} Oct 30 00:05:12.305619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.305507 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" event={"ID":"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919","Type":"ContainerStarted","Data":"33de9e4b9f1dd6e706a88e2455980c6b5eb68c3ea1736b15c2b81808a4c08354"} Oct 30 00:05:12.305619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.305538 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" event={"ID":"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919","Type":"ContainerStarted","Data":"530eb448ac7e4084e6c104fc3231a5cc9a372ce97223d50eb8f7dc20d7843c98"} Oct 30 00:05:12.307442 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.307419 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc8jd" event={"ID":"3d5c2cbc-2adb-4ad2-bb48-87071c4497db","Type":"ContainerStarted","Data":"4b6d3a5f29f6eedb992f07680611a288e803edd86a8d8d4ca6303d743f8a4c1b"} Oct 30 00:05:12.346001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.345971 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 00:05:12.346001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.345990 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 00:05:12.346210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.345994 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f68wt\"" Oct 30 00:05:12.346210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.346080 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 00:05:12.346210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.346082 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 00:05:12.346210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.346135 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 00:05:12.346421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.346400 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 00:05:12.347224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.347202 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:05:12.365405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.365379 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:12.420146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.420124 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 00:05:12.420146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.420128 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 00:05:12.420289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.420165 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 00:05:12.446888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.446866 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.446910 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.446947 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p42mx\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-kube-api-access-p42mx\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447055 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447088 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447123 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447204 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-out\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447291 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447241 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447298 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447320 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447338 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-web-config\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447468 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447424 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.447521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.447467 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.548818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548791 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-out\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.548934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548829 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.548934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548856 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.548934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548881 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.548934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548901 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-web-config\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.548934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548927 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548951 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548971 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.548990 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.549010 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p42mx\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-kube-api-access-p42mx\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.549033 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.549052 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.549078 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.549880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.549849 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.550732 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.550706 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.551987 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.551966 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-out\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.552421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.552398 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.553483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.553464 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-web-config\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.554280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.553864 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.554280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.553935 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.554280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.554087 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.554280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.554241 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.554532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.554295 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.554636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.554617 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.555469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.555437 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.573586 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.573551 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p42mx\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-kube-api-access-p42mx\") pod \"alertmanager-main-0\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.620312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.620289 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:05:12.956777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:12.956749 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:05:12.960391 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:12.960363 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdf8b73_aaee_418b_9e7b_b2e94fcda02c.slice/crio-f02d054df8edbd5012aa5c1594f6e9fada61f2fd0856692e9e504d356ce2e64f WatchSource:0}: Error finding container f02d054df8edbd5012aa5c1594f6e9fada61f2fd0856692e9e504d356ce2e64f: Status 404 returned error can't find the container with id f02d054df8edbd5012aa5c1594f6e9fada61f2fd0856692e9e504d356ce2e64f Oct 30 00:05:13.150425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.150392 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86c75469d9-wzp7z"] Oct 30 00:05:13.154956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.154583 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.171071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.171047 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 00:05:13.171178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.171087 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 00:05:13.171178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.171168 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ecjohqvakv0a7\"" Oct 30 00:05:13.171256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.171245 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9g4tn\"" Oct 30 00:05:13.173667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.173648 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 00:05:13.176401 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.176383 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86c75469d9-wzp7z"] Oct 30 00:05:13.257334 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257253 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257334 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257316 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257384 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-tmp\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257479 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-grpc-tls\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257514 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257541 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-metrics-client-ca\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257663 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-tls\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257717 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.257756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.257752 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475cm\" (UniqueName: \"kubernetes.io/projected/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-kube-api-access-475cm\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.271004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.270976 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 00:05:13.271183 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.271166 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 00:05:13.313703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.313673 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" event={"ID":"f3e8bfb3-fa1f-4af9-81ec-32c946bc7919","Type":"ContainerStarted","Data":"aeb1ef4dcab4ac28b52a7ce30ddf52c923fda401fdf64a69ddfbd335645cb601"} Oct 30 00:05:13.315400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.315371 4433 generic.go:356] "Generic (PLEG): container finished" podID="3d5c2cbc-2adb-4ad2-bb48-87071c4497db" containerID="57d6a1155947d3660de3588a6180879a60b6f6236a251903bbee6115af4c0a35" exitCode=0 Oct 30 00:05:13.315516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.315451 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc8jd" event={"ID":"3d5c2cbc-2adb-4ad2-bb48-87071c4497db","Type":"ContainerDied","Data":"57d6a1155947d3660de3588a6180879a60b6f6236a251903bbee6115af4c0a35"} Oct 30 00:05:13.316763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.316738 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerStarted","Data":"f02d054df8edbd5012aa5c1594f6e9fada61f2fd0856692e9e504d356ce2e64f"} Oct 30 00:05:13.358598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358552 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358617 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-tmp\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358662 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-grpc-tls\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358691 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358716 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-metrics-client-ca\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358751 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-tls\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358806 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358841 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475cm\" (UniqueName: \"kubernetes.io/projected/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-kube-api-access-475cm\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.358922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.358880 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.359631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.359263 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-tmp\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.360389 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.360330 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-metrics-client-ca\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.361915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.361888 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.362036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.362011 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-grpc-tls\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.363095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.363064 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-tls\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.363244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.363227 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.363296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.363262 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.364413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.364387 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.374636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.374557 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vt9fp" podStartSLOduration=2.39520017 podStartE2EDuration="3.374542428s" totalImagesPullingTime="979.342258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:13.373918881 +0000 UTC m=+22.307987435" watchObservedRunningTime="2026-10-30 00:05:13.374542428 +0000 UTC m=+22.308610965" Oct 30 00:05:13.385399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.385378 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-475cm\" (UniqueName: \"kubernetes.io/projected/f1e6fb59-2134-4a4c-acd3-fe1fa27e6851-kube-api-access-475cm\") pod \"thanos-querier-86c75469d9-wzp7z\" (UID: \"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851\") " pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.472899 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.472863 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:13.783078 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.783003 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86c75469d9-wzp7z"] Oct 30 00:05:13.785550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.785516 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:13.788697 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:13.788666 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e6fb59_2134_4a4c_acd3_fe1fa27e6851.slice/crio-dbdcf5e357e79182852fe311d39cb661bd5f4390a298a682d93ee9dea271c4cd WatchSource:0}: Error finding container dbdcf5e357e79182852fe311d39cb661bd5f4390a298a682d93ee9dea271c4cd: Status 404 returned error can't find the container with id dbdcf5e357e79182852fe311d39cb661bd5f4390a298a682d93ee9dea271c4cd Oct 30 00:05:13.896000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:13.895912 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:14.168181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.168145 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:14.277476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.277442 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:14.322189 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.322102 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" event={"ID":"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851","Type":"ContainerStarted","Data":"dbdcf5e357e79182852fe311d39cb661bd5f4390a298a682d93ee9dea271c4cd"} Oct 30 00:05:14.325366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.324842 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc8jd" event={"ID":"3d5c2cbc-2adb-4ad2-bb48-87071c4497db","Type":"ContainerStarted","Data":"e98070944c4f3b43853bee607d66b19f538a2548e7bdbb7eb6f544890987ea92"} Oct 30 00:05:14.325366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.324895 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc8jd" event={"ID":"3d5c2cbc-2adb-4ad2-bb48-87071c4497db","Type":"ContainerStarted","Data":"aa4fd7cf03f8ddc84c79ff83511752d56fa85b46b3bfb42331c0e36a11b97a9b"} Oct 30 00:05:14.326697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.326667 4433 generic.go:356] "Generic (PLEG): container finished" podID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" exitCode=0 Oct 30 00:05:14.326805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.326758 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460"} Oct 30 00:05:14.372309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.372262 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rc8jd" podStartSLOduration=3.269358445 podStartE2EDuration="4.372248872s" totalImagesPullingTime="1.020449847s" totalInitContainerRuntime="82.44058ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:14.370038043 +0000 UTC m=+23.304106591" watchObservedRunningTime="2026-10-30 00:05:14.372248872 +0000 UTC m=+23.306317407" Oct 30 00:05:14.382102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.382066 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:14.853454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:14.853410 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:15.471374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.471343 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b574c8449-xhpkt"] Oct 30 00:05:15.479932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479191 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.479932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479590 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-secret-metrics-server-tls\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.479932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479643 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm9k\" (UniqueName: \"kubernetes.io/projected/de75ec79-9b11-40ce-8523-98f3e69e814a-kube-api-access-6gm9k\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.479932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479676 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de75ec79-9b11-40ce-8523-98f3e69e814a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.479932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479743 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/de75ec79-9b11-40ce-8523-98f3e69e814a-audit-log\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.479932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479849 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-secret-metrics-server-client-certs\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.479932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479899 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/de75ec79-9b11-40ce-8523-98f3e69e814a-metrics-server-audit-profiles\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.480361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.479994 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-client-ca-bundle\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.497353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.497293 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b574c8449-xhpkt"] Oct 30 00:05:15.508662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.508099 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 00:05:15.508662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.508170 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:05:15.509290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.509270 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 00:05:15.510756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.510731 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vb94m\"" Oct 30 00:05:15.510900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.510874 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-am784q8llr7tj\"" Oct 30 00:05:15.510965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.510948 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 00:05:15.580297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.580259 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-client-ca-bundle\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.580297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.580293 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-secret-metrics-server-tls\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.580537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.580317 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm9k\" (UniqueName: \"kubernetes.io/projected/de75ec79-9b11-40ce-8523-98f3e69e814a-kube-api-access-6gm9k\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.580537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.580335 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de75ec79-9b11-40ce-8523-98f3e69e814a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.580537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.580359 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/de75ec79-9b11-40ce-8523-98f3e69e814a-audit-log\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.580537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.580378 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-secret-metrics-server-client-certs\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.580537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.580396 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/de75ec79-9b11-40ce-8523-98f3e69e814a-metrics-server-audit-profiles\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.582239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.581556 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/de75ec79-9b11-40ce-8523-98f3e69e814a-audit-log\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.582239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.582167 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de75ec79-9b11-40ce-8523-98f3e69e814a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.582424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.582259 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/de75ec79-9b11-40ce-8523-98f3e69e814a-metrics-server-audit-profiles\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.584481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.584427 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-client-ca-bundle\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.585049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.584994 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-secret-metrics-server-client-certs\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.585545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.585523 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/de75ec79-9b11-40ce-8523-98f3e69e814a-secret-metrics-server-tls\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.606000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.605976 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm9k\" (UniqueName: \"kubernetes.io/projected/de75ec79-9b11-40ce-8523-98f3e69e814a-kube-api-access-6gm9k\") pod \"metrics-server-5b574c8449-xhpkt\" (UID: \"de75ec79-9b11-40ce-8523-98f3e69e814a\") " pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.693508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.693472 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5"] Oct 30 00:05:15.695853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.695753 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:15.711206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.711181 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:15.714856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.714831 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5"] Oct 30 00:05:15.728123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.728072 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 00:05:15.728227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.728188 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-s9q8p\"" Oct 30 00:05:15.781357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.781320 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bfvl5\" (UID: \"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:15.781517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.781383 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkd9\" (UniqueName: \"kubernetes.io/projected/f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d-kube-api-access-7zkd9\") pod \"monitoring-plugin-77f4f76594-bfvl5\" (UID: \"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:15.799997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.799971 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:15.809649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.809619 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:15.882129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.882101 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bfvl5\" (UID: \"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:15.882261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.882150 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkd9\" (UniqueName: \"kubernetes.io/projected/f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d-kube-api-access-7zkd9\") pod \"monitoring-plugin-77f4f76594-bfvl5\" (UID: \"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:15.885208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.885186 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bfvl5\" (UID: \"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:15.907451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.907343 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkd9\" (UniqueName: \"kubernetes.io/projected/f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d-kube-api-access-7zkd9\") pod \"monitoring-plugin-77f4f76594-bfvl5\" (UID: \"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:15.914165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:15.914132 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:16.034171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.034097 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:16.133247 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.133217 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b574c8449-xhpkt"] Oct 30 00:05:16.136670 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:16.136640 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde75ec79_9b11_40ce_8523_98f3e69e814a.slice/crio-e16f19cc323c257760aa7c0090f374d6ef9c601b4fa3012931a6e591c4168945 WatchSource:0}: Error finding container e16f19cc323c257760aa7c0090f374d6ef9c601b4fa3012931a6e591c4168945: Status 404 returned error can't find the container with id e16f19cc323c257760aa7c0090f374d6ef9c601b4fa3012931a6e591c4168945 Oct 30 00:05:16.298769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.298741 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5c77b8459-pk9h2"] Oct 30 00:05:16.302831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.302813 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.320554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.320529 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 00:05:16.320554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.320529 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 00:05:16.320778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.320688 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 00:05:16.320778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.320694 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c55g8\"" Oct 30 00:05:16.326751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.326723 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:16.333645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.333621 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 00:05:16.333745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.333716 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 00:05:16.336084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.336050 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" event={"ID":"de75ec79-9b11-40ce-8523-98f3e69e814a","Type":"ContainerStarted","Data":"e16f19cc323c257760aa7c0090f374d6ef9c601b4fa3012931a6e591c4168945"} Oct 30 00:05:16.348468 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.348441 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c77b8459-pk9h2"] Oct 30 00:05:16.348758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.348739 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 00:05:16.350137 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.350086 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5"] Oct 30 00:05:16.386640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.386616 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-telemeter-client-tls\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.386757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.386684 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-secret-telemeter-client\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.386825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.386770 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-federate-client-tls\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.386825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.386808 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.386918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.386836 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-serving-certs-ca-bundle\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.386918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.386871 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.387040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.386992 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-metrics-client-ca\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.387094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.387058 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94v9\" (UniqueName: \"kubernetes.io/projected/b28d23f1-0da1-4611-b347-0ff9fc50e4af-kube-api-access-q94v9\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.430753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.430709 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:16.487739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.487708 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-federate-client-tls\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.487753 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.487782 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-serving-certs-ca-bundle\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.487841 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.487895 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-metrics-client-ca\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.487915 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94v9\" (UniqueName: \"kubernetes.io/projected/b28d23f1-0da1-4611-b347-0ff9fc50e4af-kube-api-access-q94v9\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.487966 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-telemeter-client-tls\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.488011 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-secret-telemeter-client\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.488543 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-serving-certs-ca-bundle\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.488949 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.488918 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-metrics-client-ca\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.489404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.489376 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28d23f1-0da1-4611-b347-0ff9fc50e4af-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.490731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.490713 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-telemeter-client-tls\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.490923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.490899 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.490979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.490920 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-federate-client-tls\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.494169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.494144 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b28d23f1-0da1-4611-b347-0ff9fc50e4af-secret-telemeter-client\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.500546 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:16.500519 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20dc4d0_e7fe_4204_9d6d_1b9ba2e20b8d.slice/crio-bb5c2955e5cd839c3a8c9a067abe0d3d1567418d30c7943ab7e543fb85d13772 WatchSource:0}: Error finding container bb5c2955e5cd839c3a8c9a067abe0d3d1567418d30c7943ab7e543fb85d13772: Status 404 returned error can't find the container with id bb5c2955e5cd839c3a8c9a067abe0d3d1567418d30c7943ab7e543fb85d13772 Oct 30 00:05:16.512744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.512721 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94v9\" (UniqueName: \"kubernetes.io/projected/b28d23f1-0da1-4611-b347-0ff9fc50e4af-kube-api-access-q94v9\") pod \"telemeter-client-5c77b8459-pk9h2\" (UID: \"b28d23f1-0da1-4611-b347-0ff9fc50e4af\") " pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:16.680515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:16.680491 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" Oct 30 00:05:17.052005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.051978 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c77b8459-pk9h2"] Oct 30 00:05:17.053826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:17.053791 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28d23f1_0da1_4611_b347_0ff9fc50e4af.slice/crio-fb86961ef03b8f81f18ff9bf46de835737659621ecfb715107b91b675e1f34c6 WatchSource:0}: Error finding container fb86961ef03b8f81f18ff9bf46de835737659621ecfb715107b91b675e1f34c6: Status 404 returned error can't find the container with id fb86961ef03b8f81f18ff9bf46de835737659621ecfb715107b91b675e1f34c6 Oct 30 00:05:17.346729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.346687 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerStarted","Data":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} Oct 30 00:05:17.346882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.346776 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerStarted","Data":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} Oct 30 00:05:17.346882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.346817 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerStarted","Data":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} Oct 30 00:05:17.346882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.346856 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerStarted","Data":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} Oct 30 00:05:17.347034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.346904 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerStarted","Data":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} Oct 30 00:05:17.348855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.348822 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" event={"ID":"b28d23f1-0da1-4611-b347-0ff9fc50e4af","Type":"ContainerStarted","Data":"fb86961ef03b8f81f18ff9bf46de835737659621ecfb715107b91b675e1f34c6"} Oct 30 00:05:17.351434 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.351363 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" event={"ID":"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d","Type":"ContainerStarted","Data":"bb5c2955e5cd839c3a8c9a067abe0d3d1567418d30c7943ab7e543fb85d13772"} Oct 30 00:05:17.355153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.354986 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" event={"ID":"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851","Type":"ContainerStarted","Data":"d32b8d6f9f256efb2102cd550cd6f5057ca006d3696aa3a2bb887392a26e1d42"} Oct 30 00:05:17.355153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.355050 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" event={"ID":"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851","Type":"ContainerStarted","Data":"8030f6531fdeac7017b049a8b0aaaf07d1f4b4d476dc46737af5adf6c4a5f2d9"} Oct 30 00:05:17.355153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.355090 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" event={"ID":"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851","Type":"ContainerStarted","Data":"7c21fc64900bfcfd3e53a9f996507c45282bc01745c3efbeac9b6411d7341577"} Oct 30 00:05:17.937822 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:17.937726 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:18.368110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:18.368021 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerStarted","Data":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} Oct 30 00:05:18.373271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:18.373236 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" event={"ID":"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851","Type":"ContainerStarted","Data":"85ff755b7b36c110c5bcddf2368b0af172047ff138e1c228aa1f32561ab0ac06"} Oct 30 00:05:18.373391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:18.373314 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" event={"ID":"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851","Type":"ContainerStarted","Data":"6971dae9326543507e2e6e57d5baa498e69109fd496f60f456a3abce619fd42e"} Oct 30 00:05:18.373391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:18.373357 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" event={"ID":"f1e6fb59-2134-4a4c-acd3-fe1fa27e6851","Type":"ContainerStarted","Data":"19cf930f0cb1d24358f97c411c27112159ff5f9bdcc0fa2f257f06127e372ce7"} Oct 30 00:05:18.373500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:18.373471 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:18.436855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:18.436798 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.546643454 podStartE2EDuration="6.436778104s" totalImagesPullingTime="3.875397297s" totalInitContainerRuntime="14.737353ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:18.436260764 +0000 UTC m=+27.370329303" watchObservedRunningTime="2026-10-30 00:05:18.436778104 +0000 UTC m=+27.370846640" Oct 30 00:05:18.487330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:18.486670 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" podStartSLOduration=1.7563594999999999 podStartE2EDuration="5.486651323s" totalImagesPullingTime="3.730291823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:18.4856595 +0000 UTC m=+27.419728040" watchObservedRunningTime="2026-10-30 00:05:18.486651323 +0000 UTC m=+27.420719858" Oct 30 00:05:19.379259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:19.379184 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" event={"ID":"de75ec79-9b11-40ce-8523-98f3e69e814a","Type":"ContainerStarted","Data":"e8e525538ed9ba5fc833c621ea66aa2ca7fc3f439ffec1166878346f6d2a48b1"} Oct 30 00:05:19.380965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:19.380926 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" event={"ID":"f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d","Type":"ContainerStarted","Data":"9479a7895330885e99e1b081adc4825cf53483bc2d65f4795d83bbb3d608e772"} Oct 30 00:05:19.381544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:19.381517 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:19.386716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:19.386689 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" Oct 30 00:05:19.423107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:19.423066 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" podStartSLOduration=2.157684229 podStartE2EDuration="4.423051957s" totalImagesPullingTime="2.265367728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:19.422522662 +0000 UTC m=+28.356591235" watchObservedRunningTime="2026-10-30 00:05:19.423051957 +0000 UTC m=+28.357120486" Oct 30 00:05:19.464357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:19.464299 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bfvl5" podStartSLOduration=2.286883091 podStartE2EDuration="4.464284002s" totalImagesPullingTime="2.177400911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:19.463392458 +0000 UTC m=+28.397460994" watchObservedRunningTime="2026-10-30 00:05:19.464284002 +0000 UTC m=+28.398352569" Oct 30 00:05:20.388209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.387665 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" event={"ID":"b28d23f1-0da1-4611-b347-0ff9fc50e4af","Type":"ContainerStarted","Data":"2e46fa12c833b423660101af380ad5c1e5818b05631a1cc9ae5455c76b79e862"} Oct 30 00:05:20.388209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.387736 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" event={"ID":"b28d23f1-0da1-4611-b347-0ff9fc50e4af","Type":"ContainerStarted","Data":"c80309d5b342ad76a8158d2ac3312e6ac5fbc76af1c89604ed5a0d3ef64dea69"} Oct 30 00:05:20.388209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.387763 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" event={"ID":"b28d23f1-0da1-4611-b347-0ff9fc50e4af","Type":"ContainerStarted","Data":"cbc6d39ac07ebae220836134138c3b7b952cd0bb80eb7c92af6c983f2aa450a4"} Oct 30 00:05:20.471134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.471077 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5c77b8459-pk9h2" podStartSLOduration=2.002258272 podStartE2EDuration="4.471059527s" totalImagesPullingTime="2.468801255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:20.470793519 +0000 UTC m=+29.404862065" watchObservedRunningTime="2026-10-30 00:05:20.471059527 +0000 UTC m=+29.405128066" Oct 30 00:05:20.475533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.475495 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:20.583451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.583374 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:20.973783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.973749 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f857b5c78-vtdpj"] Oct 30 00:05:20.987394 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:20.987362 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:21.010200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.010160 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f857b5c78-vtdpj"] Oct 30 00:05:21.010373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.010284 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032182 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 00:05:21.032354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032208 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 00:05:21.032354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032208 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-service-ca\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032259 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1905a837-ea13-4504-a397-d289bad1c6f3-tmp\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032308 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-trusted-ca-bundle\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032333 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-oauth-serving-cert\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032424 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-console-config\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032463 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-serving-cert\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032494 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkh2h\" (UniqueName: \"kubernetes.io/projected/1905a837-ea13-4504-a397-d289bad1c6f3-kube-api-access-xkh2h\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.032544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.032517 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-oauth-config\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.044849 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.044828 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 00:05:21.072617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.072543 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 00:05:21.072617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.072562 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 00:05:21.072617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.072549 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xz5vx\"" Oct 30 00:05:21.072617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.072617 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 00:05:21.101808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.101769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:21.133345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133317 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-console-config\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.133528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133354 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-serving-cert\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.133528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133381 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkh2h\" (UniqueName: \"kubernetes.io/projected/1905a837-ea13-4504-a397-d289bad1c6f3-kube-api-access-xkh2h\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.133528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133410 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-oauth-config\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.133528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133449 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-service-ca\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.133528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133467 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1905a837-ea13-4504-a397-d289bad1c6f3-tmp\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.133528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133491 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-trusted-ca-bundle\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.133528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.133506 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-oauth-serving-cert\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.134476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.134297 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-oauth-serving-cert\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.134476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.134304 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-console-config\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.134476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.134375 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-service-ca\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.134476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.134436 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1905a837-ea13-4504-a397-d289bad1c6f3-tmp\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.134745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.134685 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-trusted-ca-bundle\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.136633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.136613 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-serving-cert\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.136727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.136641 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-oauth-config\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.159332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.159310 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkh2h\" (UniqueName: \"kubernetes.io/projected/1905a837-ea13-4504-a397-d289bad1c6f3-kube-api-access-xkh2h\") pod \"console-5f857b5c78-vtdpj\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.227168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.227082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:21.332063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.332019 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:21.332310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.332290 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:21.685107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:21.685080 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f857b5c78-vtdpj"] Oct 30 00:05:21.741984 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:21.741955 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1905a837_ea13_4504_a397_d289bad1c6f3.slice/crio-845fb7b5ed16df86c26bfb9acc9c7f991debccb325c16656b154474dd0180c4c WatchSource:0}: Error finding container 845fb7b5ed16df86c26bfb9acc9c7f991debccb325c16656b154474dd0180c4c: Status 404 returned error can't find the container with id 845fb7b5ed16df86c26bfb9acc9c7f991debccb325c16656b154474dd0180c4c Oct 30 00:05:22.403225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:22.403094 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hjwsx" event={"ID":"0ef650f7-6cd3-4df0-8be4-5bc699dac77e","Type":"ContainerStarted","Data":"02207521bf6fd7e59e43eb874ceb0287a3723f893d5f5600ae29174adb9655c2"} Oct 30 00:05:22.404192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:22.404169 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:22.406947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:22.406913 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f857b5c78-vtdpj" event={"ID":"1905a837-ea13-4504-a397-d289bad1c6f3","Type":"ContainerStarted","Data":"845fb7b5ed16df86c26bfb9acc9c7f991debccb325c16656b154474dd0180c4c"} Oct 30 00:05:22.418093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:22.418048 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-hjwsx" Oct 30 00:05:22.451371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:22.451297 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-hjwsx" podStartSLOduration=2.118544138 podStartE2EDuration="20.451268497s" totalImagesPullingTime="18.332724359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:22.450708624 +0000 UTC m=+31.384777174" watchObservedRunningTime="2026-10-30 00:05:22.451268497 +0000 UTC m=+31.385337033" Oct 30 00:05:23.379901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:23.379827 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:23.520643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:23.520529 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:24.389123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:24.389092 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86c75469d9-wzp7z" Oct 30 00:05:24.900807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:24.900765 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:25.017773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:25.017733 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:25.742301 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:25.740926 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:25.850718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:25.850625 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:26.213128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:26.213058 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q9h2g" Oct 30 00:05:26.343681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:26.343643 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:26.429255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:26.429215 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f857b5c78-vtdpj" event={"ID":"1905a837-ea13-4504-a397-d289bad1c6f3","Type":"ContainerStarted","Data":"ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60"} Oct 30 00:05:26.477713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:26.477609 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f857b5c78-vtdpj" podStartSLOduration=2.653179034 podStartE2EDuration="6.477592594s" totalImagesPullingTime="3.82441356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:26.475981223 +0000 UTC m=+35.410049776" watchObservedRunningTime="2026-10-30 00:05:26.477592594 +0000 UTC m=+35.411661128" Oct 30 00:05:26.487284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:26.487251 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:31.333252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:31.333219 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:31.333758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:31.333265 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:31.337901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:31.337879 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:31.454693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:31.454666 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:05:34.335635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:34.335597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:34.472629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:34.472599 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:35.050997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:35.050960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:35.202885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:35.202844 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:35.453907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:35.453871 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:35.573532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:35.573494 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:35.800930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:35.800846 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:35.800930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:35.800906 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:37.096991 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:05:37.096955 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:05:40.731492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:40.731458 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:45.805486 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:45.805446 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:48.186108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:48.186075 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:50.458345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:50.458307 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:50.564119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:50.564081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:55.805801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:55.805771 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:55.809598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:55.809555 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b574c8449-xhpkt" Oct 30 00:05:56.457218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:56.457182 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:56.573236 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:56.573206 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:56.777912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:56.777820 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:56.888179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:56.888142 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:56.996263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:56.996234 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-567497877-228wm"] Oct 30 00:05:57.042070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.041981 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567497877-228wm"] Oct 30 00:05:57.042242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.042098 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.142389 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.142357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:57.149359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149330 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-console-config\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.149459 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149374 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-oauth-serving-cert\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.149459 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149399 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43daf76b-958f-42f5-96f4-35007fe9ae74-console-oauth-config\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.149553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149533 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-service-ca\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.149609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149563 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75pk\" (UniqueName: \"kubernetes.io/projected/43daf76b-958f-42f5-96f4-35007fe9ae74-kube-api-access-m75pk\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.149651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149605 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-trusted-ca-bundle\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.149689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149671 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43daf76b-958f-42f5-96f4-35007fe9ae74-tmp\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.149730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.149709 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43daf76b-958f-42f5-96f4-35007fe9ae74-console-serving-cert\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.250696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250663 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-console-config\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.250850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250704 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-oauth-serving-cert\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.250850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250722 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43daf76b-958f-42f5-96f4-35007fe9ae74-console-oauth-config\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.250850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250766 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-service-ca\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.250850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250782 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75pk\" (UniqueName: \"kubernetes.io/projected/43daf76b-958f-42f5-96f4-35007fe9ae74-kube-api-access-m75pk\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.250850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250799 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-trusted-ca-bundle\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.250850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250825 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43daf76b-958f-42f5-96f4-35007fe9ae74-tmp\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.251183 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.250872 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43daf76b-958f-42f5-96f4-35007fe9ae74-console-serving-cert\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.251306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.251283 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43daf76b-958f-42f5-96f4-35007fe9ae74-tmp\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.251497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.251476 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-oauth-serving-cert\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.251606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.251512 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-console-config\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.251739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.251721 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-service-ca\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.251866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.251848 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43daf76b-958f-42f5-96f4-35007fe9ae74-trusted-ca-bundle\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.253736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.253716 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43daf76b-958f-42f5-96f4-35007fe9ae74-console-oauth-config\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.253934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.253916 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43daf76b-958f-42f5-96f4-35007fe9ae74-console-serving-cert\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.257734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.257710 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:57.275494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.275469 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75pk\" (UniqueName: \"kubernetes.io/projected/43daf76b-958f-42f5-96f4-35007fe9ae74-kube-api-access-m75pk\") pod \"console-567497877-228wm\" (UID: \"43daf76b-958f-42f5-96f4-35007fe9ae74\") " pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.370235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.370204 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567497877-228wm" Oct 30 00:05:57.429790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.429761 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:57.551382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.551343 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:05:57.699475 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:57.699446 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567497877-228wm"] Oct 30 00:05:57.702138 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:05:57.702112 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43daf76b_958f_42f5_96f4_35007fe9ae74.slice/crio-5cbf291d401fbffd09afc3c7816f3a0ad5fb8ad09152ec0352e9e42cc314c018 WatchSource:0}: Error finding container 5cbf291d401fbffd09afc3c7816f3a0ad5fb8ad09152ec0352e9e42cc314c018: Status 404 returned error can't find the container with id 5cbf291d401fbffd09afc3c7816f3a0ad5fb8ad09152ec0352e9e42cc314c018 Oct 30 00:05:58.551349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:58.551315 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567497877-228wm" event={"ID":"43daf76b-958f-42f5-96f4-35007fe9ae74","Type":"ContainerStarted","Data":"be2ded974a18aa341cb68e0c3d29551f152969f4c66b90fefb6548bee8ecc6cb"} Oct 30 00:05:58.551755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:58.551394 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567497877-228wm" event={"ID":"43daf76b-958f-42f5-96f4-35007fe9ae74","Type":"ContainerStarted","Data":"5cbf291d401fbffd09afc3c7816f3a0ad5fb8ad09152ec0352e9e42cc314c018"} Oct 30 00:05:58.597279 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:05:58.597235 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-567497877-228wm" podStartSLOduration=2.597222148 podStartE2EDuration="2.597222148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:58.595904753 +0000 UTC m=+67.529973306" watchObservedRunningTime="2026-10-30 00:05:58.597222148 +0000 UTC m=+67.531290682" Oct 30 00:06:01.901724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:01.901688 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:06:01.902243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:01.902018 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="alertmanager" containerID="cri-o://30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" gracePeriod=120 Oct 30 00:06:01.902243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:01.902035 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-metric" containerID="cri-o://6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" gracePeriod=120 Oct 30 00:06:01.902243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:01.902052 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="prom-label-proxy" containerID="cri-o://ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" gracePeriod=120 Oct 30 00:06:01.902243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:01.902084 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-web" containerID="cri-o://26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" gracePeriod=120 Oct 30 00:06:01.902243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:01.902111 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy" containerID="cri-o://ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" gracePeriod=120 Oct 30 00:06:01.902243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:01.902168 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="config-reloader" containerID="cri-o://4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" gracePeriod=120 Oct 30 00:06:02.185956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.185925 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.296795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296764 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-out\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296804 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-trusted-ca-bundle\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296823 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-volume\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-volume\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296840 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-web\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296867 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296894 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-tls-assets\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-tls-assets\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296926 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-metrics-client-ca\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296951 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-main-tls\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.296977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.296977 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-main-db\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.297400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297013 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-kube-api-access-p42mx\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-kube-api-access-p42mx\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.297400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297038 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-cluster-tls-config\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.297400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297078 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.297400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297113 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-web-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-web-config\") pod \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\" (UID: \"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c\") " Oct 30 00:06:02.297400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297245 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-trusted-ca-bundle" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:02.297672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297467 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-main-db" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:06:02.297672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297625 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-metrics-client-ca" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:02.297672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297648 4433 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.297672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.297670 4433 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-alertmanager-main-db\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.299982 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.299687 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-metric" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:02.300103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.300069 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-web" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:02.300321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.300280 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-volume" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:02.300715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.300617 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-tls-assets" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:06:02.300839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.300754 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-main-tls" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:02.300839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.300767 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-out" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:06:02.301813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.301787 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:02.302520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.302490 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-kube-api-access-p42mx" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "kube-api-access-p42mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:06:02.305669 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.305567 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-cluster-tls-config" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:02.311368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.311345 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-web-config" pod "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" (UID: "0fdf8b73-aaee-418b-9e7b-b2e94fcda02c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:02.398947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398903 4433 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-web-config\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.398947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398938 4433 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-out\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.398947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398947 4433 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.398947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398958 4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.399201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398968 4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.399201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398977 4433 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-tls-assets\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.399201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398986 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-metrics-client-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.399201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.398995 4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-main-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.399201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.399005 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p42mx\" (UniqueName: \"kubernetes.io/projected/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-kube-api-access-p42mx\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.399201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.399013 4433 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-cluster-tls-config\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.399201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.399047 4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:02.571362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571332 4433 generic.go:356] "Generic (PLEG): container finished" podID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" exitCode=0 Oct 30 00:06:02.571362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571356 4433 generic.go:356] "Generic (PLEG): container finished" podID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" exitCode=0 Oct 30 00:06:02.571362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571362 4433 generic.go:356] "Generic (PLEG): container finished" podID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" exitCode=0 Oct 30 00:06:02.571362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571369 4433 generic.go:356] "Generic (PLEG): container finished" podID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" exitCode=0 Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571374 4433 generic.go:356] "Generic (PLEG): container finished" podID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" exitCode=0 Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571382 4433 generic.go:356] "Generic (PLEG): container finished" podID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" exitCode=0 Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571411 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571437 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571482 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571509 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571535 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571550 4433 scope.go:122] "RemoveContainer" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571564 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571604 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} Oct 30 00:06:02.571743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.571634 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fdf8b73-aaee-418b-9e7b-b2e94fcda02c","Type":"ContainerDied","Data":"f02d054df8edbd5012aa5c1594f6e9fada61f2fd0856692e9e504d356ce2e64f"} Oct 30 00:06:02.579515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.579495 4433 scope.go:122] "RemoveContainer" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.586364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.586346 4433 scope.go:122] "RemoveContainer" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.592759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.592743 4433 scope.go:122] "RemoveContainer" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.599072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.599058 4433 scope.go:122] "RemoveContainer" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.606020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.605991 4433 scope.go:122] "RemoveContainer" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.612629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.612606 4433 scope.go:122] "RemoveContainer" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.618937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.618912 4433 scope.go:122] "RemoveContainer" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.619179 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:02.619161 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": container with ID starting with ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374 not found: ID does not exist" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.619240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.619210 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} err="failed to get container status \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": rpc error: code = NotFound desc = could not find container \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": container with ID starting with ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374 not found: ID does not exist" Oct 30 00:06:02.619294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.619240 4433 scope.go:122] "RemoveContainer" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.619489 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:02.619470 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": container with ID starting with 6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f not found: ID does not exist" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.619529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.619496 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} err="failed to get container status \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": rpc error: code = NotFound desc = could not find container \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": container with ID starting with 6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f not found: ID does not exist" Oct 30 00:06:02.619529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.619512 4433 scope.go:122] "RemoveContainer" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.619765 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:02.619745 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": container with ID starting with ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b not found: ID does not exist" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.619809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.619769 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} err="failed to get container status \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": rpc error: code = NotFound desc = could not find container \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": container with ID starting with ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b not found: ID does not exist" Oct 30 00:06:02.619809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.619783 4433 scope.go:122] "RemoveContainer" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.619993 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:02.619977 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": container with ID starting with 26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2 not found: ID does not exist" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.620030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.619998 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} err="failed to get container status \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": rpc error: code = NotFound desc = could not find container \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": container with ID starting with 26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2 not found: ID does not exist" Oct 30 00:06:02.620030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620011 4433 scope.go:122] "RemoveContainer" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.620208 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:02.620194 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": container with ID starting with 4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9 not found: ID does not exist" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.620253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620211 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} err="failed to get container status \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": rpc error: code = NotFound desc = could not find container \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": container with ID starting with 4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9 not found: ID does not exist" Oct 30 00:06:02.620253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620222 4433 scope.go:122] "RemoveContainer" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.620404 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:02.620387 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": container with ID starting with 30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2 not found: ID does not exist" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.620440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620408 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} err="failed to get container status \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": rpc error: code = NotFound desc = could not find container \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": container with ID starting with 30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2 not found: ID does not exist" Oct 30 00:06:02.620440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620420 4433 scope.go:122] "RemoveContainer" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.620620 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:02.620602 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": container with ID starting with 03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460 not found: ID does not exist" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.620664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620623 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460"} err="failed to get container status \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": rpc error: code = NotFound desc = could not find container \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": container with ID starting with 03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460 not found: ID does not exist" Oct 30 00:06:02.620664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620634 4433 scope.go:122] "RemoveContainer" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.620817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620803 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} err="failed to get container status \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": rpc error: code = NotFound desc = could not find container \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": container with ID starting with ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374 not found: ID does not exist" Oct 30 00:06:02.620856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620817 4433 scope.go:122] "RemoveContainer" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.620997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620979 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} err="failed to get container status \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": rpc error: code = NotFound desc = could not find container \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": container with ID starting with 6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f not found: ID does not exist" Oct 30 00:06:02.621039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.620997 4433 scope.go:122] "RemoveContainer" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.621200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621184 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} err="failed to get container status \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": rpc error: code = NotFound desc = could not find container \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": container with ID starting with ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b not found: ID does not exist" Oct 30 00:06:02.621238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621200 4433 scope.go:122] "RemoveContainer" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.621390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621375 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} err="failed to get container status \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": rpc error: code = NotFound desc = could not find container \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": container with ID starting with 26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2 not found: ID does not exist" Oct 30 00:06:02.621437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621389 4433 scope.go:122] "RemoveContainer" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.621580 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621555 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} err="failed to get container status \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": rpc error: code = NotFound desc = could not find container \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": container with ID starting with 4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9 not found: ID does not exist" Oct 30 00:06:02.621623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621585 4433 scope.go:122] "RemoveContainer" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.621775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621760 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} err="failed to get container status \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": rpc error: code = NotFound desc = could not find container \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": container with ID starting with 30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2 not found: ID does not exist" Oct 30 00:06:02.621825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621774 4433 scope.go:122] "RemoveContainer" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.621952 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621937 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460"} err="failed to get container status \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": rpc error: code = NotFound desc = could not find container \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": container with ID starting with 03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460 not found: ID does not exist" Oct 30 00:06:02.621997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.621951 4433 scope.go:122] "RemoveContainer" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.622130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622114 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} err="failed to get container status \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": rpc error: code = NotFound desc = could not find container \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": container with ID starting with ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374 not found: ID does not exist" Oct 30 00:06:02.622174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622130 4433 scope.go:122] "RemoveContainer" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.622323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622307 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} err="failed to get container status \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": rpc error: code = NotFound desc = could not find container \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": container with ID starting with 6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f not found: ID does not exist" Oct 30 00:06:02.622365 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622322 4433 scope.go:122] "RemoveContainer" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.622510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622495 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} err="failed to get container status \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": rpc error: code = NotFound desc = could not find container \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": container with ID starting with ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b not found: ID does not exist" Oct 30 00:06:02.622552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622510 4433 scope.go:122] "RemoveContainer" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.622728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622709 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} err="failed to get container status \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": rpc error: code = NotFound desc = could not find container \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": container with ID starting with 26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2 not found: ID does not exist" Oct 30 00:06:02.622793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622729 4433 scope.go:122] "RemoveContainer" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.622967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622950 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} err="failed to get container status \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": rpc error: code = NotFound desc = could not find container \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": container with ID starting with 4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9 not found: ID does not exist" Oct 30 00:06:02.622967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.622965 4433 scope.go:122] "RemoveContainer" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.623152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623136 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} err="failed to get container status \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": rpc error: code = NotFound desc = could not find container \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": container with ID starting with 30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2 not found: ID does not exist" Oct 30 00:06:02.623199 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623152 4433 scope.go:122] "RemoveContainer" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.623355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623340 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460"} err="failed to get container status \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": rpc error: code = NotFound desc = could not find container \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": container with ID starting with 03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460 not found: ID does not exist" Oct 30 00:06:02.623355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623354 4433 scope.go:122] "RemoveContainer" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.623514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623498 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} err="failed to get container status \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": rpc error: code = NotFound desc = could not find container \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": container with ID starting with ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374 not found: ID does not exist" Oct 30 00:06:02.623554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623514 4433 scope.go:122] "RemoveContainer" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.623704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623689 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} err="failed to get container status \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": rpc error: code = NotFound desc = could not find container \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": container with ID starting with 6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f not found: ID does not exist" Oct 30 00:06:02.623704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623703 4433 scope.go:122] "RemoveContainer" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.623877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623860 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} err="failed to get container status \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": rpc error: code = NotFound desc = could not find container \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": container with ID starting with ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b not found: ID does not exist" Oct 30 00:06:02.623915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.623877 4433 scope.go:122] "RemoveContainer" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.624050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624034 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} err="failed to get container status \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": rpc error: code = NotFound desc = could not find container \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": container with ID starting with 26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2 not found: ID does not exist" Oct 30 00:06:02.624090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624049 4433 scope.go:122] "RemoveContainer" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.624251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624234 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} err="failed to get container status \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": rpc error: code = NotFound desc = could not find container \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": container with ID starting with 4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9 not found: ID does not exist" Oct 30 00:06:02.624298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624250 4433 scope.go:122] "RemoveContainer" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.624471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624453 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} err="failed to get container status \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": rpc error: code = NotFound desc = could not find container \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": container with ID starting with 30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2 not found: ID does not exist" Oct 30 00:06:02.624515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624474 4433 scope.go:122] "RemoveContainer" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.624749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624732 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460"} err="failed to get container status \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": rpc error: code = NotFound desc = could not find container \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": container with ID starting with 03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460 not found: ID does not exist" Oct 30 00:06:02.624749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624746 4433 scope.go:122] "RemoveContainer" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.624936 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624919 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} err="failed to get container status \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": rpc error: code = NotFound desc = could not find container \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": container with ID starting with ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374 not found: ID does not exist" Oct 30 00:06:02.624975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.624936 4433 scope.go:122] "RemoveContainer" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.625121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625106 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} err="failed to get container status \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": rpc error: code = NotFound desc = could not find container \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": container with ID starting with 6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f not found: ID does not exist" Oct 30 00:06:02.625165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625121 4433 scope.go:122] "RemoveContainer" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.625306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625292 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} err="failed to get container status \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": rpc error: code = NotFound desc = could not find container \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": container with ID starting with ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b not found: ID does not exist" Oct 30 00:06:02.625351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625306 4433 scope.go:122] "RemoveContainer" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.625508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625492 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} err="failed to get container status \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": rpc error: code = NotFound desc = could not find container \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": container with ID starting with 26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2 not found: ID does not exist" Oct 30 00:06:02.625543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625508 4433 scope.go:122] "RemoveContainer" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.625722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625705 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} err="failed to get container status \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": rpc error: code = NotFound desc = could not find container \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": container with ID starting with 4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9 not found: ID does not exist" Oct 30 00:06:02.625767 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625722 4433 scope.go:122] "RemoveContainer" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.625904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625888 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} err="failed to get container status \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": rpc error: code = NotFound desc = could not find container \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": container with ID starting with 30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2 not found: ID does not exist" Oct 30 00:06:02.625945 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.625903 4433 scope.go:122] "RemoveContainer" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.626098 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626083 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460"} err="failed to get container status \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": rpc error: code = NotFound desc = could not find container \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": container with ID starting with 03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460 not found: ID does not exist" Oct 30 00:06:02.626142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626097 4433 scope.go:122] "RemoveContainer" containerID="ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374" Oct 30 00:06:02.626280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626266 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374"} err="failed to get container status \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": rpc error: code = NotFound desc = could not find container \"ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374\": container with ID starting with ab0148fba4ee4cbb79d9acd0db6f8d4b2b799d0377420fe13a92462a46918374 not found: ID does not exist" Oct 30 00:06:02.626325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626280 4433 scope.go:122] "RemoveContainer" containerID="6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f" Oct 30 00:06:02.626463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626448 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f"} err="failed to get container status \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": rpc error: code = NotFound desc = could not find container \"6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f\": container with ID starting with 6cbc12a6be394461cd0c69b13756c42223c126a18d1b57a351caf522c8c0a18f not found: ID does not exist" Oct 30 00:06:02.626463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626462 4433 scope.go:122] "RemoveContainer" containerID="ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b" Oct 30 00:06:02.626661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626643 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b"} err="failed to get container status \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": rpc error: code = NotFound desc = could not find container \"ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b\": container with ID starting with ca40ce7b5c86459c04a249e40b55d15f13e157211957eff831e6c6763bba683b not found: ID does not exist" Oct 30 00:06:02.626705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626661 4433 scope.go:122] "RemoveContainer" containerID="26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2" Oct 30 00:06:02.626853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626835 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2"} err="failed to get container status \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": rpc error: code = NotFound desc = could not find container \"26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2\": container with ID starting with 26b0d8fcae2747b4372af1fea80677410c35f328e080e5bd0cea165f65e920c2 not found: ID does not exist" Oct 30 00:06:02.626896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.626854 4433 scope.go:122] "RemoveContainer" containerID="4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9" Oct 30 00:06:02.627088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.627073 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9"} err="failed to get container status \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": rpc error: code = NotFound desc = could not find container \"4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9\": container with ID starting with 4480633e770aabf5897a8f69b21661245339cc31b1fa7f24bfde9c403ba9f0d9 not found: ID does not exist" Oct 30 00:06:02.627088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.627086 4433 scope.go:122] "RemoveContainer" containerID="30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2" Oct 30 00:06:02.627270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.627254 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2"} err="failed to get container status \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": rpc error: code = NotFound desc = could not find container \"30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2\": container with ID starting with 30da577579d1fe9e812b7a003f9b6dc579e99452f276cc47951ba82f2183c4f2 not found: ID does not exist" Oct 30 00:06:02.627307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.627270 4433 scope.go:122] "RemoveContainer" containerID="03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460" Oct 30 00:06:02.627456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.627442 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460"} err="failed to get container status \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": rpc error: code = NotFound desc = could not find container \"03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460\": container with ID starting with 03f3951d3fc1aa81d7270bac76e6fcb828c38cf0a97dc44a9ecd4fd504f97460 not found: ID does not exist" Oct 30 00:06:02.631398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.631380 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:06:02.635657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.635639 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:06:02.665482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665460 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:06:02.665772 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665760 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="alertmanager" Oct 30 00:06:02.665812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665773 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="alertmanager" Oct 30 00:06:02.665812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665788 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="prom-label-proxy" Oct 30 00:06:02.665812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665793 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="prom-label-proxy" Oct 30 00:06:02.665812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665802 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="init-config-reloader" Oct 30 00:06:02.665812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665807 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="init-config-reloader" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665818 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-web" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665823 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-web" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665830 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665834 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665841 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-metric" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665845 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-metric" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665854 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="config-reloader" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665858 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="config-reloader" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665903 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-web" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665909 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665915 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="kube-rbac-proxy-metric" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665921 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="config-reloader" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665929 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="alertmanager" Oct 30 00:06:02.665957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.665935 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" containerName="prom-label-proxy" Oct 30 00:06:02.699795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.699770 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.702507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.702482 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:06:02.716482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716451 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 00:06:02.716482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716451 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 00:06:02.716662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716492 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 00:06:02.716662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716461 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 00:06:02.716662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716461 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 00:06:02.716662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716547 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 00:06:02.716662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716661 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 00:06:02.716910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716664 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 00:06:02.716910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.716664 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f68wt\"" Oct 30 00:06:02.729418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.729398 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 00:06:02.803511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803482 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df2499fe-2768-4988-9f24-4815a7d93fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803527 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803549 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803631 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df2499fe-2768-4988-9f24-4815a7d93fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803764 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803795 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803824 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803866 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.803991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803911 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df2499fe-2768-4988-9f24-4815a7d93fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.804079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.803995 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df2499fe-2768-4988-9f24-4815a7d93fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.804079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.804039 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.804150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.804088 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df2499fe-2768-4988-9f24-4815a7d93fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.804150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.804107 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmkz\" (UniqueName: \"kubernetes.io/projected/df2499fe-2768-4988-9f24-4815a7d93fbc-kube-api-access-5zmkz\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.904931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.904842 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.904931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.904881 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.904931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.904903 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df2499fe-2768-4988-9f24-4815a7d93fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.904931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.904929 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.904948 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.904976 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905000 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905116 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df2499fe-2768-4988-9f24-4815a7d93fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905168 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df2499fe-2768-4988-9f24-4815a7d93fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905200 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905227 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df2499fe-2768-4988-9f24-4815a7d93fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905254 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmkz\" (UniqueName: \"kubernetes.io/projected/df2499fe-2768-4988-9f24-4815a7d93fbc-kube-api-access-5zmkz\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.905600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905280 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df2499fe-2768-4988-9f24-4815a7d93fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.906060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.905716 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df2499fe-2768-4988-9f24-4815a7d93fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.907587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.906185 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df2499fe-2768-4988-9f24-4815a7d93fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.907587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.907517 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df2499fe-2768-4988-9f24-4815a7d93fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.908420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.908035 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df2499fe-2768-4988-9f24-4815a7d93fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.908420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.908267 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.908420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.908378 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.908420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.908382 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.908420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.908412 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.908740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.908485 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.908740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.908679 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.909286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.909263 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df2499fe-2768-4988-9f24-4815a7d93fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.909473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.909456 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df2499fe-2768-4988-9f24-4815a7d93fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:02.930211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:02.930190 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmkz\" (UniqueName: \"kubernetes.io/projected/df2499fe-2768-4988-9f24-4815a7d93fbc-kube-api-access-5zmkz\") pod \"alertmanager-main-0\" (UID: \"df2499fe-2768-4988-9f24-4815a7d93fbc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:03.113929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:03.113888 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:06:03.180276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:03.180236 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fdf8b73-aaee-418b-9e7b-b2e94fcda02c" path="/var/lib/kubelet/pods/0fdf8b73-aaee-418b-9e7b-b2e94fcda02c/volumes" Oct 30 00:06:03.456335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:03.456312 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:06:03.458477 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:06:03.458450 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2499fe_2768_4988_9f24_4815a7d93fbc.slice/crio-288b4078d259662603629e605ebd15bd1b05145a079889abef131327399bd329 WatchSource:0}: Error finding container 288b4078d259662603629e605ebd15bd1b05145a079889abef131327399bd329: Status 404 returned error can't find the container with id 288b4078d259662603629e605ebd15bd1b05145a079889abef131327399bd329 Oct 30 00:06:03.582545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:03.582508 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"e2c15b842cf13be743c6ced2ad2b88a468d30da9752783f88987f4dfc568e2cd"} Oct 30 00:06:03.582672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:03.582602 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"288b4078d259662603629e605ebd15bd1b05145a079889abef131327399bd329"} Oct 30 00:06:04.590185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590152 4433 generic.go:356] "Generic (PLEG): container finished" podID="df2499fe-2768-4988-9f24-4815a7d93fbc" containerID="e2c15b842cf13be743c6ced2ad2b88a468d30da9752783f88987f4dfc568e2cd" exitCode=0 Oct 30 00:06:04.590720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590208 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerDied","Data":"e2c15b842cf13be743c6ced2ad2b88a468d30da9752783f88987f4dfc568e2cd"} Oct 30 00:06:04.590720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590257 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"b53701e84b1146a8b9eb0d68bfae6f15bfe5bfa67cfc54d4202e41c7e1e7fae2"} Oct 30 00:06:04.590720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590282 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"c8d94d35262ec8d990b357987512eae357b596da2f02dc9f21901559ee31cb55"} Oct 30 00:06:04.590720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590306 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"6b8bd804428be4ac2addb946e68e0f0c13d0e0144a4638c44ec0c9d68eae72be"} Oct 30 00:06:04.590720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590328 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"7742e658e4ef0d99fef2cff03da232ec67a7c64f57f66c022db8e71c4da616fc"} Oct 30 00:06:04.590720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590350 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"18d8e8af62f2c76652a1da3e57c860397bcd004bc24c9ab600531e9613fce26b"} Oct 30 00:06:04.590720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.590372 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df2499fe-2768-4988-9f24-4815a7d93fbc","Type":"ContainerStarted","Data":"d05107166769a1919e19b284cfb5fb4f96aa0a0496850c5d6b04232bdb833777"} Oct 30 00:06:04.664150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:04.664082 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.649573069 podStartE2EDuration="2.66406349s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.490421ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:06:04.664016829 +0000 UTC m=+73.598085363" watchObservedRunningTime="2026-10-30 00:06:04.66406349 +0000 UTC m=+73.598132023" Oct 30 00:06:07.097379 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:07.097337 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:06:07.371362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:07.371280 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-567497877-228wm" Oct 30 00:06:07.371362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:07.371354 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-567497877-228wm" Oct 30 00:06:07.376009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:07.375986 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-567497877-228wm" Oct 30 00:06:07.613834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:07.613805 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-567497877-228wm" Oct 30 00:06:07.687964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:07.687892 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f857b5c78-vtdpj"] Oct 30 00:06:07.795467 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:07.795426 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:06:07.901956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:07.901893 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:06:20.482357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:20.482317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:06:20.592463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:20.592428 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:06:26.085404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:26.085372 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:06:32.707561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:32.707505 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f857b5c78-vtdpj" podUID="1905a837-ea13-4504-a397-d289bad1c6f3" containerName="console" containerID="cri-o://ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60" gracePeriod=15 Oct 30 00:06:32.949331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:32.949304 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:06:33.053340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053260 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-service-ca\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-service-ca\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053295 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1905a837-ea13-4504-a397-d289bad1c6f3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1905a837-ea13-4504-a397-d289bad1c6f3-tmp\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053317 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-oauth-config\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053348 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-trusted-ca-bundle\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053372 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-console-config\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-console-config\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053402 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-oauth-serving-cert\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053427 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1905a837-ea13-4504-a397-d289bad1c6f3-kube-api-access-xkh2h\" (UniqueName: \"kubernetes.io/projected/1905a837-ea13-4504-a397-d289bad1c6f3-kube-api-access-xkh2h\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053459 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-serving-cert\") pod \"1905a837-ea13-4504-a397-d289bad1c6f3\" (UID: \"1905a837-ea13-4504-a397-d289bad1c6f3\") " Oct 30 00:06:33.053655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053568 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905a837-ea13-4504-a397-d289bad1c6f3-tmp" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:06:33.053923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053724 4433 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1905a837-ea13-4504-a397-d289bad1c6f3-tmp\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.053923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053805 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-trusted-ca-bundle" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:33.053923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.053728 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-service-ca" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:33.054045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.054023 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-oauth-serving-cert" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:33.054082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.054068 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-console-config" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:33.055750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.055723 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1905a837-ea13-4504-a397-d289bad1c6f3-kube-api-access-xkh2h" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "kube-api-access-xkh2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:06:33.055966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.055947 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-oauth-config" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:33.056034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.055970 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-serving-cert" pod "1905a837-ea13-4504-a397-d289bad1c6f3" (UID: "1905a837-ea13-4504-a397-d289bad1c6f3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:06:33.154583 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.154536 4433 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-console-config\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.154583 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.154561 4433 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-oauth-serving-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.154777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.154597 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkh2h\" (UniqueName: \"kubernetes.io/projected/1905a837-ea13-4504-a397-d289bad1c6f3-kube-api-access-xkh2h\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.154777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.154607 4433 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-serving-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.154777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.154616 4433 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.154777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.154623 4433 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1905a837-ea13-4504-a397-d289bad1c6f3-console-oauth-config\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.154777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.154631 4433 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1905a837-ea13-4504-a397-d289bad1c6f3-trusted-ca-bundle\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:06:33.724049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.724014 4433 generic.go:356] "Generic (PLEG): container finished" podID="1905a837-ea13-4504-a397-d289bad1c6f3" containerID="ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60" exitCode=0 Oct 30 00:06:33.724472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.724060 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f857b5c78-vtdpj" Oct 30 00:06:33.724472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.724098 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f857b5c78-vtdpj" event={"ID":"1905a837-ea13-4504-a397-d289bad1c6f3","Type":"ContainerDied","Data":"ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60"} Oct 30 00:06:33.724472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.724146 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f857b5c78-vtdpj" event={"ID":"1905a837-ea13-4504-a397-d289bad1c6f3","Type":"ContainerDied","Data":"845fb7b5ed16df86c26bfb9acc9c7f991debccb325c16656b154474dd0180c4c"} Oct 30 00:06:33.724472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.724166 4433 scope.go:122] "RemoveContainer" containerID="ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60" Oct 30 00:06:33.732143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.732126 4433 scope.go:122] "RemoveContainer" containerID="ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60" Oct 30 00:06:33.732385 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:33.732366 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60\": container with ID starting with ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60 not found: ID does not exist" containerID="ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60" Oct 30 00:06:33.732452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.732392 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60"} err="failed to get container status \"ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60\": rpc error: code = NotFound desc = could not find container \"ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60\": container with ID starting with ea370c7b831d83bc387e321a292f97403711ddc952dfbb65930d548f4461fe60 not found: ID does not exist" Oct 30 00:06:33.776869 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.776843 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f857b5c78-vtdpj"] Oct 30 00:06:33.790271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:33.790252 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f857b5c78-vtdpj"] Oct 30 00:06:35.173798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:35.173767 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:06:35.176779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:06:35.176755 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1905a837-ea13-4504-a397-d289bad1c6f3" path="/var/lib/kubelet/pods/1905a837-ea13-4504-a397-d289bad1c6f3/volumes" Oct 30 00:06:37.111956 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:06:37.111924 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:07:07.098153 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:07:07.098071 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:07:33.040731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:07:33.040693 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:07:35.298020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:07:35.297992 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:07:37.096667 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:07:37.096633 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:07:43.181148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:07:43.181117 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:07:43.281362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:07:43.281329 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:07.096995 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:08:07.096957 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:08:16.093625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:16.093593 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:27.628342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:27.628293 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:27.761369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:27.761335 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:27.893664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:27.893584 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:29.302513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.302469 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl"] Oct 30 00:08:29.302928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.302830 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1905a837-ea13-4504-a397-d289bad1c6f3" containerName="console" Oct 30 00:08:29.302928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.302845 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1905a837-ea13-4504-a397-d289bad1c6f3" containerName="console" Oct 30 00:08:29.302928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.302911 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1905a837-ea13-4504-a397-d289bad1c6f3" containerName="console" Oct 30 00:08:29.304735 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.304719 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.318771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.318746 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:29.320862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.320844 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 00:08:29.320862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.320848 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 00:08:29.320990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.320886 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nht5w\"" Oct 30 00:08:29.321076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.321062 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 00:08:29.321247 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.321229 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:08:29.321331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.321303 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:08:29.321331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.321313 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:08:29.335983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.335960 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl"] Oct 30 00:08:29.405980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.405952 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.406127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.405996 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.406127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.406017 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.406223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.406163 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.406223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.406192 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.406223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.406213 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nqc\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-kube-api-access-m6nqc\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.406334 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.406234 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.430437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.430411 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:29.507567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.507538 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nqc\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-kube-api-access-m6nqc\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.507714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.507589 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.507714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.507624 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.507714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.507653 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.507889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.507780 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.507889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.507850 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.507889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.507879 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.508537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.508515 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.510085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.510057 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.510190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.510126 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.510403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.510383 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.510438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.510409 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.532117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.532090 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.532246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.532229 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nqc\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-kube-api-access-m6nqc\") pod \"istiod-openshift-gateway-7876bfcb74-cbprl\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.614452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.614422 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:29.929376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:29.929302 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl"] Oct 30 00:08:29.932435 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:08:29.932408 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5a1df9_f5cc_492b_adbe_92e28030b6d5.slice/crio-7d19fac309fb8a85c286b581b3d5b3c5b6e83a050c4b08f2a3d9c9fec586e282 WatchSource:0}: Error finding container 7d19fac309fb8a85c286b581b3d5b3c5b6e83a050c4b08f2a3d9c9fec586e282: Status 404 returned error can't find the container with id 7d19fac309fb8a85c286b581b3d5b3c5b6e83a050c4b08f2a3d9c9fec586e282 Oct 30 00:08:30.168031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:30.167992 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" event={"ID":"1b5a1df9-f5cc-492b-adbe-92e28030b6d5","Type":"ContainerStarted","Data":"7d19fac309fb8a85c286b581b3d5b3c5b6e83a050c4b08f2a3d9c9fec586e282"} Oct 30 00:08:30.624960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:30.624924 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:31.118784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:31.118748 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:31.234217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:31.234174 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:32.428011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:32.427979 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:08:32.428259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:32.428047 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:08:33.146535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.146493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:33.182481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.182451 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" event={"ID":"1b5a1df9-f5cc-492b-adbe-92e28030b6d5","Type":"ContainerStarted","Data":"422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672"} Oct 30 00:08:33.182674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.182646 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:33.184207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.184177 4433 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cbprl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:08:33.184321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.184225 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" podUID="1b5a1df9-f5cc-492b-adbe-92e28030b6d5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:08:33.251154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.251104 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" podStartSLOduration=1.7582570560000002 podStartE2EDuration="4.251092661s" totalImagesPullingTime="2.492835605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:08:33.250519873 +0000 UTC m=+222.184588407" watchObservedRunningTime="2026-10-30 00:08:33.251092661 +0000 UTC m=+222.185161195" Oct 30 00:08:33.259306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.259272 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:33.959391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:33.959363 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:34.075148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:34.075112 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:34.181169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:34.181136 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:34.186515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:34.186489 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:08:35.444018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:35.443946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:37.065124 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:37.065097 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:37.097029 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:08:37.096997 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:08:47.802086 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:47.802035 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:57.448445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:57.448408 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:57.615343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:57.615298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:57.731708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:57.731625 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:58.311235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.311195 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:58.487371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.487337 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k"] Oct 30 00:08:58.497434 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.497412 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.507519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.507492 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k"] Oct 30 00:08:58.558329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.558297 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.558329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.558330 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a2f5d22-d526-48b6-8151-738abb326a92-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.558560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.558349 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmz8w\" (UniqueName: \"kubernetes.io/projected/2a2f5d22-d526-48b6-8151-738abb326a92-kube-api-access-dmz8w\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.558560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.558393 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.558560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.558479 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a2f5d22-d526-48b6-8151-738abb326a92-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.558560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.558497 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.591011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.590937 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:58.601583 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.601553 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:08:58.601713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.601598 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 00:08:58.601713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.601610 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-84fdv\"" Oct 30 00:08:58.601713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.601598 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 00:08:58.601713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.601629 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:08:58.601713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.601656 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:08:58.659109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.659080 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.659332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.659119 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a2f5d22-d526-48b6-8151-738abb326a92-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.659332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.659144 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmz8w\" (UniqueName: \"kubernetes.io/projected/2a2f5d22-d526-48b6-8151-738abb326a92-kube-api-access-dmz8w\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.659332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.659251 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.659332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.659309 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a2f5d22-d526-48b6-8151-738abb326a92-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.659554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.659341 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.659554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.659466 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a2f5d22-d526-48b6-8151-738abb326a92-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.661511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.661486 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a2f5d22-d526-48b6-8151-738abb326a92-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.661784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.661766 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.661874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.661845 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.661874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.661863 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a2f5d22-d526-48b6-8151-738abb326a92-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.685046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.685021 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmz8w\" (UniqueName: \"kubernetes.io/projected/2a2f5d22-d526-48b6-8151-738abb326a92-kube-api-access-dmz8w\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k\" (UID: \"2a2f5d22-d526-48b6-8151-738abb326a92\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.700259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.700234 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:58.814419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.814385 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:08:58.861649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.861608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:58.973476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:58.973421 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:08:59.180598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:59.180556 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k"] Oct 30 00:08:59.182875 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:08:59.182844 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2f5d22_d526_48b6_8151_738abb326a92.slice/crio-2a023b61883b8ef25ccb38eeddfa10a4caa4a60ee83ef8a31a4dc460af3e46df WatchSource:0}: Error finding container 2a023b61883b8ef25ccb38eeddfa10a4caa4a60ee83ef8a31a4dc460af3e46df: Status 404 returned error can't find the container with id 2a023b61883b8ef25ccb38eeddfa10a4caa4a60ee83ef8a31a4dc460af3e46df Oct 30 00:08:59.272647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:08:59.272609 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" event={"ID":"2a2f5d22-d526-48b6-8151-738abb326a92","Type":"ContainerStarted","Data":"2a023b61883b8ef25ccb38eeddfa10a4caa4a60ee83ef8a31a4dc460af3e46df"} Oct 30 00:09:01.028683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:01.028593 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:01.153125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:01.153087 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:02.287840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:02.287768 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" event={"ID":"2a2f5d22-d526-48b6-8151-738abb326a92","Type":"ContainerStarted","Data":"829b382b30905845e9bbca0c5e71592e826ab0b00bc4d467710b0b8880a6bad0"} Oct 30 00:09:03.293257 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:03.293221 4433 generic.go:356] "Generic (PLEG): container finished" podID="2a2f5d22-d526-48b6-8151-738abb326a92" containerID="829b382b30905845e9bbca0c5e71592e826ab0b00bc4d467710b0b8880a6bad0" exitCode=0 Oct 30 00:09:03.293658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:03.293262 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" event={"ID":"2a2f5d22-d526-48b6-8151-738abb326a92","Type":"ContainerDied","Data":"829b382b30905845e9bbca0c5e71592e826ab0b00bc4d467710b0b8880a6bad0"} Oct 30 00:09:04.299882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:04.299842 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" event={"ID":"2a2f5d22-d526-48b6-8151-738abb326a92","Type":"ContainerStarted","Data":"b2615834ed2c15a306bb0507fdf9cd214d8c32fe455831524c792037284e1ec3"} Oct 30 00:09:04.300309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:04.300013 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:09:04.392633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:04.392550 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" podStartSLOduration=2.84956166 podStartE2EDuration="6.39253685s" totalImagesPullingTime="3.121417271s" totalInitContainerRuntime="421.557919ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:04.377533368 +0000 UTC m=+253.311601915" watchObservedRunningTime="2026-10-30 00:09:04.39253685 +0000 UTC m=+253.326605385" Oct 30 00:09:05.231044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:05.231011 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:05.347052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:05.347007 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:07.096350 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:09:07.096316 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:09:15.305974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:15.305943 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-qjh5k" Oct 30 00:09:22.043942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:22.043891 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:22.940013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:22.939977 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:26.217771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:26.217733 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:26.336437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:26.336393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:26.796800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:26.796765 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:26.901930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:26.901898 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:28.883289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:28.883232 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:29.071637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:29.071602 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:31.302108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:31.302073 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:31.644506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:31.644465 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:33.430665 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:33.430632 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:33.544561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:33.544528 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:35.620728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:35.620677 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:36.386110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:36.386076 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:37.097111 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:09:37.097070 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:09:37.603129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:37.603095 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:38.389737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:38.389702 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:38.498324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:38.498283 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:38.605850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:38.605818 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:39.280166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:39.280133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:43.509113 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:43.509066 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:43.654309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:43.654271 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:43.801540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:43.801453 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:45.788889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:45.788857 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:46.036730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:46.036697 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:47.496034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:47.496000 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:47.610968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:47.610874 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:48.202366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:48.202323 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:49.632272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:49.632222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:50.936790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:50.936756 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:51.060139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.060089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:51.195666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.195589 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:09:51.195909 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.195887 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:09:51.309974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.309942 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:51.635435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.635397 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw"] Oct 30 00:09:51.641306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.641282 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.645928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.645887 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:51.672160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.672134 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw"] Oct 30 00:09:51.701676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.701651 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6bbvl\"" Oct 30 00:09:51.701853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.701748 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 00:09:51.701853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.701847 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 00:09:51.793684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.793648 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27fad148-1b28-4abc-9bb0-11a38d22f28e-tmp\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.793899 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.793695 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmn4\" (UniqueName: \"kubernetes.io/projected/27fad148-1b28-4abc-9bb0-11a38d22f28e-kube-api-access-dtmn4\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.793899 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.793816 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fad148-1b28-4abc-9bb0-11a38d22f28e-tls-certs\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.834340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.834303 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:51.894922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.894844 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fad148-1b28-4abc-9bb0-11a38d22f28e-tls-certs\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.895084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.894937 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27fad148-1b28-4abc-9bb0-11a38d22f28e-tmp\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.895084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.894973 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmn4\" (UniqueName: \"kubernetes.io/projected/27fad148-1b28-4abc-9bb0-11a38d22f28e-kube-api-access-dtmn4\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.897262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.897226 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27fad148-1b28-4abc-9bb0-11a38d22f28e-tmp\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.897504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.897484 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fad148-1b28-4abc-9bb0-11a38d22f28e-tls-certs\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.927190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.927159 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmn4\" (UniqueName: \"kubernetes.io/projected/27fad148-1b28-4abc-9bb0-11a38d22f28e-kube-api-access-dtmn4\") pod \"kube-auth-proxy-64d88dd7bf-th2sw\" (UID: \"27fad148-1b28-4abc-9bb0-11a38d22f28e\") " pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.953705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.953675 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" Oct 30 00:09:51.968694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:51.968651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:52.079330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:52.079299 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:52.252621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:52.252474 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw"] Oct 30 00:09:52.254734 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:09:52.254702 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fad148_1b28_4abc_9bb0_11a38d22f28e.slice/crio-4fbd3f7b5f2799df0d7de27a77c501fda40a45474d472ea826011b44d638bbf9 WatchSource:0}: Error finding container 4fbd3f7b5f2799df0d7de27a77c501fda40a45474d472ea826011b44d638bbf9: Status 404 returned error can't find the container with id 4fbd3f7b5f2799df0d7de27a77c501fda40a45474d472ea826011b44d638bbf9 Oct 30 00:09:52.492717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:52.492663 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" event={"ID":"27fad148-1b28-4abc-9bb0-11a38d22f28e","Type":"ContainerStarted","Data":"4fbd3f7b5f2799df0d7de27a77c501fda40a45474d472ea826011b44d638bbf9"} Oct 30 00:09:53.320059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:53.320005 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:54.261508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:54.261473 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:54.376335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:54.376295 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:54.494342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:54.494306 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:54.600750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:54.600712 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:54.707093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:54.707062 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:54.996309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:54.996233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:55.100719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:55.100686 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:55.507249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:55.507210 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" event={"ID":"27fad148-1b28-4abc-9bb0-11a38d22f28e","Type":"ContainerStarted","Data":"b858a1950dc09b15c20dcd6e6c7c48393290022eb50144d163a5b7d3af91f3ee"} Oct 30 00:09:55.560824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:55.560739 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64d88dd7bf-th2sw" podStartSLOduration=2.18958353 podStartE2EDuration="4.560724255s" totalImagesPullingTime="2.371140725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:55.559552664 +0000 UTC m=+304.493621199" watchObservedRunningTime="2026-10-30 00:09:55.560724255 +0000 UTC m=+304.494792810" Oct 30 00:09:56.594560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:56.594512 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:09:56.699926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:09:56.699897 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:00.869120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:00.869080 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:00.985922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:00.985885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:05.675324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:05.675232 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:05.783097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:05.783062 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:07.096864 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:10:07.096819 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:10:10.407297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:10.407260 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:10.522740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:10.522706 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:15.609519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:15.609472 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:15.719351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:15.719322 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:16.010307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:16.010200 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:16.122410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:16.122378 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:16.238796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:16.238762 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:16.420114 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:16.420077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:16.527046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:16.527008 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:17.115548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.115511 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp"] Oct 30 00:10:17.118935 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.118910 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.138494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.138468 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp"] Oct 30 00:10:17.150492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.150468 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:10:17.153206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.153186 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 00:10:17.153295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.153193 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 00:10:17.153295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.153236 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 00:10:17.153295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.153263 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 00:10:17.153406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.153338 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-m62wr\"" Oct 30 00:10:17.214712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.214683 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6714f311-dfc3-4727-b9b8-018915d05567-manager-config\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.214888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.214741 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfc4\" (UniqueName: \"kubernetes.io/projected/6714f311-dfc3-4727-b9b8-018915d05567-kube-api-access-sdfc4\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.214888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.214819 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6714f311-dfc3-4727-b9b8-018915d05567-cert\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.214888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.214852 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6714f311-dfc3-4727-b9b8-018915d05567-metrics-certs\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.315926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.315893 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6714f311-dfc3-4727-b9b8-018915d05567-manager-config\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.316094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.315951 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfc4\" (UniqueName: \"kubernetes.io/projected/6714f311-dfc3-4727-b9b8-018915d05567-kube-api-access-sdfc4\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.316094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.315988 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6714f311-dfc3-4727-b9b8-018915d05567-cert\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.316094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.316009 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6714f311-dfc3-4727-b9b8-018915d05567-metrics-certs\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.316621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.316596 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6714f311-dfc3-4727-b9b8-018915d05567-manager-config\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.318493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.318463 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6714f311-dfc3-4727-b9b8-018915d05567-cert\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.318642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.318621 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6714f311-dfc3-4727-b9b8-018915d05567-metrics-certs\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.340224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.340195 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfc4\" (UniqueName: \"kubernetes.io/projected/6714f311-dfc3-4727-b9b8-018915d05567-kube-api-access-sdfc4\") pod \"jobset-controller-manager-5ffdccb548-wsvmp\" (UID: \"6714f311-dfc3-4727-b9b8-018915d05567\") " pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.430321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.430246 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:17.728107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.727996 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp"] Oct 30 00:10:17.730793 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:10:17.730765 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6714f311_dfc3_4727_b9b8_018915d05567.slice/crio-595ae20a2331a6132238485f327169d6152ea05137c9caf3b7dcdd2ef8159b93 WatchSource:0}: Error finding container 595ae20a2331a6132238485f327169d6152ea05137c9caf3b7dcdd2ef8159b93: Status 404 returned error can't find the container with id 595ae20a2331a6132238485f327169d6152ea05137c9caf3b7dcdd2ef8159b93 Oct 30 00:10:17.732782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:17.732761 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:10:18.598546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:18.598509 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" event={"ID":"6714f311-dfc3-4727-b9b8-018915d05567","Type":"ContainerStarted","Data":"595ae20a2331a6132238485f327169d6152ea05137c9caf3b7dcdd2ef8159b93"} Oct 30 00:10:20.609122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:20.609082 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" event={"ID":"6714f311-dfc3-4727-b9b8-018915d05567","Type":"ContainerStarted","Data":"360beb2bfd320f233e7e11c260abd1b83e6a625fbe6937d9139843409f26a54a"} Oct 30 00:10:20.609520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:20.609276 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:20.653695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:20.653647 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" podStartSLOduration=0.910123375 podStartE2EDuration="3.653635158s" totalImagesPullingTime="2.743511783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:20.653153144 +0000 UTC m=+329.587221697" watchObservedRunningTime="2026-10-30 00:10:20.653635158 +0000 UTC m=+329.587703693" Oct 30 00:10:23.200375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:23.200339 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:23.312320 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:23.312286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:27.215730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:27.215697 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:27.334253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:27.334220 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:31.618902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:31.618876 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:10:32.048661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:32.048556 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:32.181744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:32.181707 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:35.705851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:35.705817 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:37.097232 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:10:37.097198 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:10:37.577448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:37.577411 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:37.690523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:37.690482 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:37.801776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:37.801743 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:37.906378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:37.906290 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:38.505625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:38.505593 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:38.618972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:38.618939 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:53.535816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.535771 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:53.908909 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.908874 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w2kzg"] Oct 30 00:10:53.911206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.911186 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:53.929418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.929391 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w2kzg"] Oct 30 00:10:53.946798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.946762 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-cl84z\"" Oct 30 00:10:53.946798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.946784 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:10:53.947003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.946814 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:10:53.947003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:53.946824 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 00:10:54.017325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.017285 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ecbf8c-f8a8-4cfa-884c-d066b189b266-tls-certs\") pod \"model-serving-api-75bd89fdfc-w2kzg\" (UID: \"79ecbf8c-f8a8-4cfa-884c-d066b189b266\") " pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:54.017494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.017349 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66prg\" (UniqueName: \"kubernetes.io/projected/79ecbf8c-f8a8-4cfa-884c-d066b189b266-kube-api-access-66prg\") pod \"model-serving-api-75bd89fdfc-w2kzg\" (UID: \"79ecbf8c-f8a8-4cfa-884c-d066b189b266\") " pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:54.117983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.117947 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ecbf8c-f8a8-4cfa-884c-d066b189b266-tls-certs\") pod \"model-serving-api-75bd89fdfc-w2kzg\" (UID: \"79ecbf8c-f8a8-4cfa-884c-d066b189b266\") " pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:54.118145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.117999 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66prg\" (UniqueName: \"kubernetes.io/projected/79ecbf8c-f8a8-4cfa-884c-d066b189b266-kube-api-access-66prg\") pod \"model-serving-api-75bd89fdfc-w2kzg\" (UID: \"79ecbf8c-f8a8-4cfa-884c-d066b189b266\") " pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:54.120508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.120481 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ecbf8c-f8a8-4cfa-884c-d066b189b266-tls-certs\") pod \"model-serving-api-75bd89fdfc-w2kzg\" (UID: \"79ecbf8c-f8a8-4cfa-884c-d066b189b266\") " pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:54.141977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.141947 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66prg\" (UniqueName: \"kubernetes.io/projected/79ecbf8c-f8a8-4cfa-884c-d066b189b266-kube-api-access-66prg\") pod \"model-serving-api-75bd89fdfc-w2kzg\" (UID: \"79ecbf8c-f8a8-4cfa-884c-d066b189b266\") " pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:54.222743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.222678 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:54.519283 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.519258 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w2kzg"] Oct 30 00:10:54.521689 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:10:54.521658 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ecbf8c_f8a8_4cfa_884c_d066b189b266.slice/crio-c162fc9f0e7c197c8375eae3260c6f56407d97c8d9b00ec8fdab4ea57a8dd10f WatchSource:0}: Error finding container c162fc9f0e7c197c8375eae3260c6f56407d97c8d9b00ec8fdab4ea57a8dd10f: Status 404 returned error can't find the container with id c162fc9f0e7c197c8375eae3260c6f56407d97c8d9b00ec8fdab4ea57a8dd10f Oct 30 00:10:54.758015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:54.757982 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" event={"ID":"79ecbf8c-f8a8-4cfa-884c-d066b189b266","Type":"ContainerStarted","Data":"c162fc9f0e7c197c8375eae3260c6f56407d97c8d9b00ec8fdab4ea57a8dd10f"} Oct 30 00:10:55.603205 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.603168 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw"] Oct 30 00:10:55.606755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.606733 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:55.625862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.625837 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw"] Oct 30 00:10:55.631233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.631205 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44e25d2a-4611-4532-b4db-c28d758ee4f1-data\") pod \"postgres-7cdbf6dc44-vwjgw\" (UID: \"44e25d2a-4611-4532-b4db-c28d758ee4f1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:55.631391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.631270 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kt5q\" (UniqueName: \"kubernetes.io/projected/44e25d2a-4611-4532-b4db-c28d758ee4f1-kube-api-access-4kt5q\") pod \"postgres-7cdbf6dc44-vwjgw\" (UID: \"44e25d2a-4611-4532-b4db-c28d758ee4f1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:55.637165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.637137 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:10:55.637274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.637206 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:10:55.637274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.637230 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 00:10:55.639840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.639817 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6j2kz\"" Oct 30 00:10:55.732079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.732045 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44e25d2a-4611-4532-b4db-c28d758ee4f1-data\") pod \"postgres-7cdbf6dc44-vwjgw\" (UID: \"44e25d2a-4611-4532-b4db-c28d758ee4f1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:55.732241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.732115 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kt5q\" (UniqueName: \"kubernetes.io/projected/44e25d2a-4611-4532-b4db-c28d758ee4f1-kube-api-access-4kt5q\") pod \"postgres-7cdbf6dc44-vwjgw\" (UID: \"44e25d2a-4611-4532-b4db-c28d758ee4f1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:55.732386 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.732364 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44e25d2a-4611-4532-b4db-c28d758ee4f1-data\") pod \"postgres-7cdbf6dc44-vwjgw\" (UID: \"44e25d2a-4611-4532-b4db-c28d758ee4f1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:55.756658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.756628 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kt5q\" (UniqueName: \"kubernetes.io/projected/44e25d2a-4611-4532-b4db-c28d758ee4f1-kube-api-access-4kt5q\") pod \"postgres-7cdbf6dc44-vwjgw\" (UID: \"44e25d2a-4611-4532-b4db-c28d758ee4f1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:55.922493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:55.922414 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:10:56.004860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.004806 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:56.223850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.223828 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw"] Oct 30 00:10:56.226340 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:10:56.226311 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e25d2a_4611_4532_b4db_c28d758ee4f1.slice/crio-8311f72ace0be9badf75061b9772c2e9293c87a0700dc35f37743a5ce59acd72 WatchSource:0}: Error finding container 8311f72ace0be9badf75061b9772c2e9293c87a0700dc35f37743a5ce59acd72: Status 404 returned error can't find the container with id 8311f72ace0be9badf75061b9772c2e9293c87a0700dc35f37743a5ce59acd72 Oct 30 00:10:56.249829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.249792 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:56.355076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.355043 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:56.461105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.461001 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:56.567910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.567874 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:56.672553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.672517 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:56.768075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.767970 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" event={"ID":"79ecbf8c-f8a8-4cfa-884c-d066b189b266","Type":"ContainerStarted","Data":"2c458295ca6dc3bd1e7c42c37eb0f00117b1912228a905e6e98025496466d95e"} Oct 30 00:10:56.768240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.768088 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:10:56.769117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.769089 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" event={"ID":"44e25d2a-4611-4532-b4db-c28d758ee4f1","Type":"ContainerStarted","Data":"8311f72ace0be9badf75061b9772c2e9293c87a0700dc35f37743a5ce59acd72"} Oct 30 00:10:56.810608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:56.810542 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" podStartSLOduration=2.19442184 podStartE2EDuration="3.810530363s" totalImagesPullingTime="1.616108523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:56.809943065 +0000 UTC m=+365.744011614" watchObservedRunningTime="2026-10-30 00:10:56.810530363 +0000 UTC m=+365.744598899" Oct 30 00:10:57.479220 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.479070 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl"] Oct 30 00:10:57.480063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.479319 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" podUID="1b5a1df9-f5cc-492b-adbe-92e28030b6d5" containerName="discovery" containerID="cri-o://422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672" gracePeriod=30 Oct 30 00:10:57.736902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.736856 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:10:57.762296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.762260 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:10:57.775453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.775418 4433 generic.go:356] "Generic (PLEG): container finished" podID="1b5a1df9-f5cc-492b-adbe-92e28030b6d5" containerID="422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672" exitCode=0 Oct 30 00:10:57.776049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.776026 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" Oct 30 00:10:57.776178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.776147 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" event={"ID":"1b5a1df9-f5cc-492b-adbe-92e28030b6d5","Type":"ContainerDied","Data":"422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672"} Oct 30 00:10:57.776240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.776186 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl" event={"ID":"1b5a1df9-f5cc-492b-adbe-92e28030b6d5","Type":"ContainerDied","Data":"7d19fac309fb8a85c286b581b3d5b3c5b6e83a050c4b08f2a3d9c9fec586e282"} Oct 30 00:10:57.776240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.776213 4433 scope.go:122] "RemoveContainer" containerID="422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672" Oct 30 00:10:57.789716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.789681 4433 scope.go:122] "RemoveContainer" containerID="422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672" Oct 30 00:10:57.790289 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:10:57.790238 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672\": container with ID starting with 422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672 not found: ID does not exist" containerID="422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672" Oct 30 00:10:57.790407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.790278 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672"} err="failed to get container status \"422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672\": rpc error: code = NotFound desc = could not find container \"422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672\": container with ID starting with 422edddcc97c9e2fcda299051b4d44e86a13474c5cd5b7498305d34a35f55672 not found: ID does not exist" Oct 30 00:10:57.852207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.852172 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-kubeconfig\") pod \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " Oct 30 00:10:57.852373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.852241 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-cacerts\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-cacerts\") pod \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " Oct 30 00:10:57.852373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.852270 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-token\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-token\") pod \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " Oct 30 00:10:57.852373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.852334 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-kube-api-access-m6nqc\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-kube-api-access-m6nqc\") pod \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " Oct 30 00:10:57.852480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.852388 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-ca-configmap\") pod \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " Oct 30 00:10:57.852480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.852423 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-dns-cert\") pod \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " Oct 30 00:10:57.852480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.852462 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-local-certs\") pod \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\" (UID: \"1b5a1df9-f5cc-492b-adbe-92e28030b6d5\") " Oct 30 00:10:57.853698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.853666 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-ca-configmap" pod "1b5a1df9-f5cc-492b-adbe-92e28030b6d5" (UID: "1b5a1df9-f5cc-492b-adbe-92e28030b6d5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:10:57.858272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.858236 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-dns-cert" pod "1b5a1df9-f5cc-492b-adbe-92e28030b6d5" (UID: "1b5a1df9-f5cc-492b-adbe-92e28030b6d5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:10:57.858650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.858501 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-local-certs" pod "1b5a1df9-f5cc-492b-adbe-92e28030b6d5" (UID: "1b5a1df9-f5cc-492b-adbe-92e28030b6d5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:10:57.858650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.858622 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-token" pod "1b5a1df9-f5cc-492b-adbe-92e28030b6d5" (UID: "1b5a1df9-f5cc-492b-adbe-92e28030b6d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:10:57.858827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.858755 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-kubeconfig" pod "1b5a1df9-f5cc-492b-adbe-92e28030b6d5" (UID: "1b5a1df9-f5cc-492b-adbe-92e28030b6d5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:10:57.859288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.859251 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-cacerts" pod "1b5a1df9-f5cc-492b-adbe-92e28030b6d5" (UID: "1b5a1df9-f5cc-492b-adbe-92e28030b6d5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:10:57.872711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.872673 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-kube-api-access-m6nqc" pod "1b5a1df9-f5cc-492b-adbe-92e28030b6d5" (UID: "1b5a1df9-f5cc-492b-adbe-92e28030b6d5"). InnerVolumeSpecName "kube-api-access-m6nqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:10:57.953470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.953432 4433 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-cacerts\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:10:57.953645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.953495 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:10:57.953645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.953509 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nqc\" (UniqueName: \"kubernetes.io/projected/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-kube-api-access-m6nqc\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:10:57.953645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.953522 4433 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-ca-configmap\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:10:57.953645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.953536 4433 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-csr-dns-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:10:57.953645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.953545 4433 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-local-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:10:57.953645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:57.953554 4433 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b5a1df9-f5cc-492b-adbe-92e28030b6d5-istio-kubeconfig\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:10:58.125977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:58.125938 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl"] Oct 30 00:10:58.139778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:58.139739 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbprl"] Oct 30 00:10:59.178801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:10:59.178771 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5a1df9-f5cc-492b-adbe-92e28030b6d5" path="/var/lib/kubelet/pods/1b5a1df9-f5cc-492b-adbe-92e28030b6d5/volumes" Oct 30 00:11:01.801311 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:01.801213 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" event={"ID":"44e25d2a-4611-4532-b4db-c28d758ee4f1","Type":"ContainerStarted","Data":"fb6d5c677989dc1d4250e538add48ddd793054bd750ee9ed7dfcac492a1e0f13"} Oct 30 00:11:01.801826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:01.801341 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:11:01.843138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:01.843088 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" podStartSLOduration=1.627970082 podStartE2EDuration="6.843063104s" totalImagesPullingTime="5.215093022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:11:01.842953849 +0000 UTC m=+370.777022389" watchObservedRunningTime="2026-10-30 00:11:01.843063104 +0000 UTC m=+370.777131635" Oct 30 00:11:07.096945 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:11:07.096907 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:11:07.780129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:07.780101 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-w2kzg" Oct 30 00:11:07.842842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:07.842812 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vwjgw" Oct 30 00:11:15.097996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.097959 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:15.177739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.177706 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b6fdcd695-8qpwb"] Oct 30 00:11:15.178242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.178226 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1b5a1df9-f5cc-492b-adbe-92e28030b6d5" containerName="discovery" Oct 30 00:11:15.178294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.178244 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5a1df9-f5cc-492b-adbe-92e28030b6d5" containerName="discovery" Oct 30 00:11:15.178354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.178342 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5a1df9-f5cc-492b-adbe-92e28030b6d5" containerName="discovery" Oct 30 00:11:15.180685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.180664 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.195139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.195110 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6fdcd695-8qpwb"] Oct 30 00:11:15.204740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.204703 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:15.207101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.207078 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 00:11:15.207211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.207101 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 00:11:15.207211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.207102 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 00:11:15.207211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.207102 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q75hf\"" Oct 30 00:11:15.307088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.307056 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbds\" (UniqueName: \"kubernetes.io/projected/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-kube-api-access-cnbds\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.307281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.307099 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-metrics-certs\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.307281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.307133 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-webhook-cert\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.312915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.312855 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:15.408382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.408291 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbds\" (UniqueName: \"kubernetes.io/projected/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-kube-api-access-cnbds\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.408382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.408342 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-metrics-certs\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.408382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.408378 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-webhook-cert\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.410996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.410962 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-metrics-certs\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.411110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.410962 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-webhook-cert\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.416496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.416466 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:15.432527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.432500 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbds\" (UniqueName: \"kubernetes.io/projected/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-kube-api-access-cnbds\") pod \"maas-controller-5b6fdcd695-8qpwb\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.492498 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.492464 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:15.706441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.706360 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77f7bd4b46-9mpx5"] Oct 30 00:11:15.708874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.708851 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.728884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.728859 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f7bd4b46-9mpx5"] Oct 30 00:11:15.794145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.794124 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6fdcd695-8qpwb"] Oct 30 00:11:15.796692 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:11:15.796661 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20362ed7_6913_4254_b8d7_8f1c5d13d4c0.slice/crio-02252bd4e1cf51403cc9f8340b1d3d25ab9ebd68a6348b15c7d373aaeac18b46 WatchSource:0}: Error finding container 02252bd4e1cf51403cc9f8340b1d3d25ab9ebd68a6348b15c7d373aaeac18b46: Status 404 returned error can't find the container with id 02252bd4e1cf51403cc9f8340b1d3d25ab9ebd68a6348b15c7d373aaeac18b46 Oct 30 00:11:15.812161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.812139 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ba5f5f-a605-4090-82c2-6265d249c20b-metrics-certs\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.812161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.812167 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04ba5f5f-a605-4090-82c2-6265d249c20b-webhook-cert\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.812312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.812196 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8l9v\" (UniqueName: \"kubernetes.io/projected/04ba5f5f-a605-4090-82c2-6265d249c20b-kube-api-access-j8l9v\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.863666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.863633 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" event={"ID":"20362ed7-6913-4254-b8d7-8f1c5d13d4c0","Type":"ContainerStarted","Data":"02252bd4e1cf51403cc9f8340b1d3d25ab9ebd68a6348b15c7d373aaeac18b46"} Oct 30 00:11:15.913088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.913057 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ba5f5f-a605-4090-82c2-6265d249c20b-metrics-certs\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.913088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.913090 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04ba5f5f-a605-4090-82c2-6265d249c20b-webhook-cert\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.913286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.913109 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8l9v\" (UniqueName: \"kubernetes.io/projected/04ba5f5f-a605-4090-82c2-6265d249c20b-kube-api-access-j8l9v\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.915555 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.915530 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ba5f5f-a605-4090-82c2-6265d249c20b-metrics-certs\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.915673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.915608 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04ba5f5f-a605-4090-82c2-6265d249c20b-webhook-cert\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:15.936870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:15.936839 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8l9v\" (UniqueName: \"kubernetes.io/projected/04ba5f5f-a605-4090-82c2-6265d249c20b-kube-api-access-j8l9v\") pod \"maas-controller-77f7bd4b46-9mpx5\" (UID: \"04ba5f5f-a605-4090-82c2-6265d249c20b\") " pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:16.021009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:16.020934 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:16.321125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:16.321093 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f7bd4b46-9mpx5"] Oct 30 00:11:16.323194 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:11:16.323164 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ba5f5f_a605_4090_82c2_6265d249c20b.slice/crio-5bacd99e8563c6df629b3ce36b311803c116d2732045d4fdf7504bfe142942e3 WatchSource:0}: Error finding container 5bacd99e8563c6df629b3ce36b311803c116d2732045d4fdf7504bfe142942e3: Status 404 returned error can't find the container with id 5bacd99e8563c6df629b3ce36b311803c116d2732045d4fdf7504bfe142942e3 Oct 30 00:11:16.872642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:16.872592 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" event={"ID":"04ba5f5f-a605-4090-82c2-6265d249c20b","Type":"ContainerStarted","Data":"5bacd99e8563c6df629b3ce36b311803c116d2732045d4fdf7504bfe142942e3"} Oct 30 00:11:18.597790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.597753 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:18.705222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.705132 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:18.885530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.885475 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" event={"ID":"04ba5f5f-a605-4090-82c2-6265d249c20b","Type":"ContainerStarted","Data":"e16d2b1fc9807c03c98402b30687a1ddebbfd7cce72cbfd470c982f609c2dd59"} Oct 30 00:11:18.885720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.885610 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:18.887016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.886993 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" event={"ID":"20362ed7-6913-4254-b8d7-8f1c5d13d4c0","Type":"ContainerStarted","Data":"0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1"} Oct 30 00:11:18.887153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.887049 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:18.931953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.931890 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" podStartSLOduration=1.896109505 podStartE2EDuration="3.931873092s" totalImagesPullingTime="2.035763587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:11:18.931521444 +0000 UTC m=+387.865589979" watchObservedRunningTime="2026-10-30 00:11:18.931873092 +0000 UTC m=+387.865941630" Oct 30 00:11:18.978410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:18.978316 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" podStartSLOduration=1.419796114 podStartE2EDuration="3.978302346s" totalImagesPullingTime="2.558506232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:11:18.977543572 +0000 UTC m=+387.911612117" watchObservedRunningTime="2026-10-30 00:11:18.978302346 +0000 UTC m=+387.912370881" Oct 30 00:11:20.477567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.477531 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp"] Oct 30 00:11:20.480013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.479991 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.498823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.498772 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-sqq2j\"" Oct 30 00:11:20.512156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.512130 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 00:11:20.512156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.512166 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 00:11:20.512156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.512183 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 00:11:20.513961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.513941 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp"] Oct 30 00:11:20.553480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.553445 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cldcx\" (UniqueName: \"kubernetes.io/projected/9c408603-395a-45b8-8154-da48848349dc-kube-api-access-cldcx\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.553662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.553497 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c408603-395a-45b8-8154-da48848349dc-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.553717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.553694 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9c408603-395a-45b8-8154-da48848349dc-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.553753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.553729 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9c408603-395a-45b8-8154-da48848349dc-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.564727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.564694 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:20.654568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.654531 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c408603-395a-45b8-8154-da48848349dc-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.654818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.654628 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9c408603-395a-45b8-8154-da48848349dc-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.654818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.654651 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9c408603-395a-45b8-8154-da48848349dc-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.654818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.654671 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cldcx\" (UniqueName: \"kubernetes.io/projected/9c408603-395a-45b8-8154-da48848349dc-kube-api-access-cldcx\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.655385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.655360 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c408603-395a-45b8-8154-da48848349dc-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.655506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.655488 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9c408603-395a-45b8-8154-da48848349dc-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.656933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.656914 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9c408603-395a-45b8-8154-da48848349dc-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.670098 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.670063 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:20.678433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.678412 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cldcx\" (UniqueName: \"kubernetes.io/projected/9c408603-395a-45b8-8154-da48848349dc-kube-api-access-cldcx\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-h5kbp\" (UID: \"9c408603-395a-45b8-8154-da48848349dc\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:20.791325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:20.791241 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:21.086036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:21.086013 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp"] Oct 30 00:11:21.088826 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:11:21.088796 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c408603_395a_45b8_8154_da48848349dc.slice/crio-9a6e9d0ee73260cabe49f3a7e28f4b04895e92a789db8252705b8119dceae705 WatchSource:0}: Error finding container 9a6e9d0ee73260cabe49f3a7e28f4b04895e92a789db8252705b8119dceae705: Status 404 returned error can't find the container with id 9a6e9d0ee73260cabe49f3a7e28f4b04895e92a789db8252705b8119dceae705 Oct 30 00:11:21.900496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:21.900456 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" event={"ID":"9c408603-395a-45b8-8154-da48848349dc","Type":"ContainerStarted","Data":"9a6e9d0ee73260cabe49f3a7e28f4b04895e92a789db8252705b8119dceae705"} Oct 30 00:11:29.893834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:29.893799 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77f7bd4b46-9mpx5" Oct 30 00:11:29.894753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:29.894734 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:29.966449 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:29.966414 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b6fdcd695-8qpwb"] Oct 30 00:11:29.966663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:29.966609 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" podUID="20362ed7-6913-4254-b8d7-8f1c5d13d4c0" containerName="manager" containerID="cri-o://0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1" gracePeriod=30 Oct 30 00:11:30.212306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.212275 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:30.349993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.349961 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-webhook-cert\") pod \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " Oct 30 00:11:30.350178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.350000 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-metrics-certs\") pod \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " Oct 30 00:11:30.350178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.350142 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-kube-api-access-cnbds\" (UniqueName: \"kubernetes.io/projected/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-kube-api-access-cnbds\") pod \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\" (UID: \"20362ed7-6913-4254-b8d7-8f1c5d13d4c0\") " Oct 30 00:11:30.352599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.352551 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-kube-api-access-cnbds" pod "20362ed7-6913-4254-b8d7-8f1c5d13d4c0" (UID: "20362ed7-6913-4254-b8d7-8f1c5d13d4c0"). InnerVolumeSpecName "kube-api-access-cnbds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:11:30.352718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.352621 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-webhook-cert" pod "20362ed7-6913-4254-b8d7-8f1c5d13d4c0" (UID: "20362ed7-6913-4254-b8d7-8f1c5d13d4c0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:11:30.352771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.352735 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-metrics-certs" pod "20362ed7-6913-4254-b8d7-8f1c5d13d4c0" (UID: "20362ed7-6913-4254-b8d7-8f1c5d13d4c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:11:30.451116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.451027 4433 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-webhook-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:11:30.451116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.451071 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:11:30.451116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.451089 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbds\" (UniqueName: \"kubernetes.io/projected/20362ed7-6913-4254-b8d7-8f1c5d13d4c0-kube-api-access-cnbds\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:11:30.952952 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.952919 4433 generic.go:356] "Generic (PLEG): container finished" podID="20362ed7-6913-4254-b8d7-8f1c5d13d4c0" containerID="0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1" exitCode=0 Oct 30 00:11:30.953508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.952971 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" Oct 30 00:11:30.953508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.952974 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" event={"ID":"20362ed7-6913-4254-b8d7-8f1c5d13d4c0","Type":"ContainerDied","Data":"0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1"} Oct 30 00:11:30.953508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.953107 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6fdcd695-8qpwb" event={"ID":"20362ed7-6913-4254-b8d7-8f1c5d13d4c0","Type":"ContainerDied","Data":"02252bd4e1cf51403cc9f8340b1d3d25ab9ebd68a6348b15c7d373aaeac18b46"} Oct 30 00:11:30.953508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.953137 4433 scope.go:122] "RemoveContainer" containerID="0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1" Oct 30 00:11:30.962679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.962660 4433 scope.go:122] "RemoveContainer" containerID="0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1" Oct 30 00:11:30.962991 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:11:30.962969 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1\": container with ID starting with 0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1 not found: ID does not exist" containerID="0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1" Oct 30 00:11:30.963033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:30.963002 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1"} err="failed to get container status \"0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1\": rpc error: code = NotFound desc = could not find container \"0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1\": container with ID starting with 0e33f7fadf413a5fa5c44f9c860a4ed26183ef86effcd83bb007c75fcb6070b1 not found: ID does not exist" Oct 30 00:11:31.006676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:31.006643 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b6fdcd695-8qpwb"] Oct 30 00:11:31.020163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:31.020131 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b6fdcd695-8qpwb"] Oct 30 00:11:31.178350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:31.178311 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20362ed7-6913-4254-b8d7-8f1c5d13d4c0" path="/var/lib/kubelet/pods/20362ed7-6913-4254-b8d7-8f1c5d13d4c0/volumes" Oct 30 00:11:33.972699 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:33.972651 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" event={"ID":"9c408603-395a-45b8-8154-da48848349dc","Type":"ContainerStarted","Data":"3f5ef5f67652787a6dd19d26364560b7b30e140d8f3afc8ea3899836b73ca704"} Oct 30 00:11:33.973202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:33.972798 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:34.016328 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:34.016279 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" podStartSLOduration=2.085536406 podStartE2EDuration="14.016265903s" totalImagesPullingTime="11.930729497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:11:34.014595939 +0000 UTC m=+402.948664471" watchObservedRunningTime="2026-10-30 00:11:34.016265903 +0000 UTC m=+402.950334437" Oct 30 00:11:35.830031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:35.829946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:37.096515 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:11:37.096475 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:11:44.982194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:44.982120 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-h5kbp" Oct 30 00:11:49.144610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:49.144551 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:49.253670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:49.253634 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:50.532926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.532862 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:50.635932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.635902 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:50.708361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.708326 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d789f6877-84tfz"] Oct 30 00:11:50.708819 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.708800 4433 cpu_manager.go:414] "containerMap: removing container" podUID="20362ed7-6913-4254-b8d7-8f1c5d13d4c0" containerName="manager" Oct 30 00:11:50.708931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.708820 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20362ed7-6913-4254-b8d7-8f1c5d13d4c0" containerName="manager" Oct 30 00:11:50.708931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.708908 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="20362ed7-6913-4254-b8d7-8f1c5d13d4c0" containerName="manager" Oct 30 00:11:50.713584 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.713549 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.730662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.730637 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d789f6877-84tfz"] Oct 30 00:11:50.743686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.743655 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:50.747514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.747490 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 00:11:50.747639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.747626 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w2hx9\"" Oct 30 00:11:50.747726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.747710 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 00:11:50.797695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.797622 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4cbca3-f11b-4658-a33d-3485d45988c8-openshift-service-ca\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.797695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.797660 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-metrics-certs\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.797864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.797774 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-maas-api-tls\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.797864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.797799 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8m8l\" (UniqueName: \"kubernetes.io/projected/4b4cbca3-f11b-4658-a33d-3485d45988c8-kube-api-access-v8m8l\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.898733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.898700 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-maas-api-tls\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.898733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.898734 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8m8l\" (UniqueName: \"kubernetes.io/projected/4b4cbca3-f11b-4658-a33d-3485d45988c8-kube-api-access-v8m8l\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.898964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.898774 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4cbca3-f11b-4658-a33d-3485d45988c8-openshift-service-ca\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.898964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.898800 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-metrics-certs\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.899494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.899468 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4cbca3-f11b-4658-a33d-3485d45988c8-openshift-service-ca\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.901226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.901201 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-maas-api-tls\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.901407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.901385 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-metrics-certs\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:50.922712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:50.922686 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8m8l\" (UniqueName: \"kubernetes.io/projected/4b4cbca3-f11b-4658-a33d-3485d45988c8-kube-api-access-v8m8l\") pod \"maas-api-d789f6877-84tfz\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:51.025757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:51.025719 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:51.323919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:51.323893 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d789f6877-84tfz"] Oct 30 00:11:51.326142 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:11:51.326111 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4cbca3_f11b_4658_a33d_3485d45988c8.slice/crio-6f8509f4add526bee6f2c530af0bd5aa0a19c70650f80563cf7539e0d4f42a5f WatchSource:0}: Error finding container 6f8509f4add526bee6f2c530af0bd5aa0a19c70650f80563cf7539e0d4f42a5f: Status 404 returned error can't find the container with id 6f8509f4add526bee6f2c530af0bd5aa0a19c70650f80563cf7539e0d4f42a5f Oct 30 00:11:52.047379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:52.047339 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" event={"ID":"4b4cbca3-f11b-4658-a33d-3485d45988c8","Type":"ContainerStarted","Data":"6f8509f4add526bee6f2c530af0bd5aa0a19c70650f80563cf7539e0d4f42a5f"} Oct 30 00:11:53.052866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:53.052827 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" event={"ID":"4b4cbca3-f11b-4658-a33d-3485d45988c8","Type":"ContainerStarted","Data":"f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237"} Oct 30 00:11:53.053211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:53.052994 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:53.102307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:53.102254 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" podStartSLOduration=1.456052883 podStartE2EDuration="3.102239567s" totalImagesPullingTime="1.646186684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:11:53.101066414 +0000 UTC m=+422.035134959" watchObservedRunningTime="2026-10-30 00:11:53.102239567 +0000 UTC m=+422.036308102" Oct 30 00:11:59.062595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:59.062539 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:11:59.554448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:59.554364 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:11:59.756500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:11:59.756452 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:12:00.054080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.054047 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66d7794744-s8p8s"] Oct 30 00:12:00.069042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.069004 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.078312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.078286 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d7794744-s8p8s"] Oct 30 00:12:00.164623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.164585 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhv9n\" (UniqueName: \"kubernetes.io/projected/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-kube-api-access-rhv9n\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.164623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.164626 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-metrics-certs\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.164830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.164702 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-maas-api-tls\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.164830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.164771 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-openshift-service-ca\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.265460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.265421 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhv9n\" (UniqueName: \"kubernetes.io/projected/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-kube-api-access-rhv9n\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.265460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.265462 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-metrics-certs\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.265702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.265506 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-maas-api-tls\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.265702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.265533 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-openshift-service-ca\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.266446 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.266421 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-openshift-service-ca\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.268074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.268045 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-metrics-certs\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.268178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.268160 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-maas-api-tls\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.289413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.289388 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhv9n\" (UniqueName: \"kubernetes.io/projected/6f13b10d-a3d2-4e2f-83ff-f74e51e7849b-kube-api-access-rhv9n\") pod \"maas-api-66d7794744-s8p8s\" (UID: \"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b\") " pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.383564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.383537 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:00.678048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:00.677997 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d7794744-s8p8s"] Oct 30 00:12:00.680314 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:12:00.680289 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f13b10d_a3d2_4e2f_83ff_f74e51e7849b.slice/crio-9fffe8378315176bcdf2e0c7cc1702f3af644b7143e94ecab622bf5e0bbad3ed WatchSource:0}: Error finding container 9fffe8378315176bcdf2e0c7cc1702f3af644b7143e94ecab622bf5e0bbad3ed: Status 404 returned error can't find the container with id 9fffe8378315176bcdf2e0c7cc1702f3af644b7143e94ecab622bf5e0bbad3ed Oct 30 00:12:01.091008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:01.090979 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" event={"ID":"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b","Type":"ContainerStarted","Data":"96adcc9510af687f260b9e163a4eab0699acff18c2c99e07e86cb6b42f08afac"} Oct 30 00:12:01.091384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:01.091034 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" event={"ID":"6f13b10d-a3d2-4e2f-83ff-f74e51e7849b","Type":"ContainerStarted","Data":"9fffe8378315176bcdf2e0c7cc1702f3af644b7143e94ecab622bf5e0bbad3ed"} Oct 30 00:12:01.091384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:01.091079 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:01.136450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:01.136391 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" podStartSLOduration=0.808459779 podStartE2EDuration="1.136378286s" totalImagesPullingTime="327.918507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:01.135224381 +0000 UTC m=+430.069292932" watchObservedRunningTime="2026-10-30 00:12:01.136378286 +0000 UTC m=+430.070446821" Oct 30 00:12:07.097486 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:12:07.097447 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:12:07.100762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.100736 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66d7794744-s8p8s" Oct 30 00:12:07.178744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.178717 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d789f6877-84tfz"] Oct 30 00:12:07.178964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.178931 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" podUID="4b4cbca3-f11b-4658-a33d-3485d45988c8" containerName="maas-api" containerID="cri-o://f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237" gracePeriod=30 Oct 30 00:12:07.439400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.439371 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:12:07.521854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.521813 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-maas-api-tls\") pod \"4b4cbca3-f11b-4658-a33d-3485d45988c8\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " Oct 30 00:12:07.522042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.521898 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4cbca3-f11b-4658-a33d-3485d45988c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4cbca3-f11b-4658-a33d-3485d45988c8-openshift-service-ca\") pod \"4b4cbca3-f11b-4658-a33d-3485d45988c8\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " Oct 30 00:12:07.522042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.521937 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-metrics-certs\") pod \"4b4cbca3-f11b-4658-a33d-3485d45988c8\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " Oct 30 00:12:07.522042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.521975 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4cbca3-f11b-4658-a33d-3485d45988c8-kube-api-access-v8m8l\" (UniqueName: \"kubernetes.io/projected/4b4cbca3-f11b-4658-a33d-3485d45988c8-kube-api-access-v8m8l\") pod \"4b4cbca3-f11b-4658-a33d-3485d45988c8\" (UID: \"4b4cbca3-f11b-4658-a33d-3485d45988c8\") " Oct 30 00:12:07.522272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.522246 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4cbca3-f11b-4658-a33d-3485d45988c8-openshift-service-ca" pod "4b4cbca3-f11b-4658-a33d-3485d45988c8" (UID: "4b4cbca3-f11b-4658-a33d-3485d45988c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:12:07.524092 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.524059 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4cbca3-f11b-4658-a33d-3485d45988c8-kube-api-access-v8m8l" pod "4b4cbca3-f11b-4658-a33d-3485d45988c8" (UID: "4b4cbca3-f11b-4658-a33d-3485d45988c8"). InnerVolumeSpecName "kube-api-access-v8m8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:12:07.524092 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.524077 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-metrics-certs" pod "4b4cbca3-f11b-4658-a33d-3485d45988c8" (UID: "4b4cbca3-f11b-4658-a33d-3485d45988c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:12:07.524228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.524098 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-maas-api-tls" pod "4b4cbca3-f11b-4658-a33d-3485d45988c8" (UID: "4b4cbca3-f11b-4658-a33d-3485d45988c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:12:07.623390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.623312 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4cbca3-f11b-4658-a33d-3485d45988c8-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:12:07.623390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.623351 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:12:07.623390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.623365 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8m8l\" (UniqueName: \"kubernetes.io/projected/4b4cbca3-f11b-4658-a33d-3485d45988c8-kube-api-access-v8m8l\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:12:07.623390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:07.623380 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4cbca3-f11b-4658-a33d-3485d45988c8-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:12:08.123168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.123134 4433 generic.go:356] "Generic (PLEG): container finished" podID="4b4cbca3-f11b-4658-a33d-3485d45988c8" containerID="f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237" exitCode=0 Oct 30 00:12:08.123559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.123184 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" Oct 30 00:12:08.123559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.123224 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" event={"ID":"4b4cbca3-f11b-4658-a33d-3485d45988c8","Type":"ContainerDied","Data":"f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237"} Oct 30 00:12:08.123559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.123308 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d789f6877-84tfz" event={"ID":"4b4cbca3-f11b-4658-a33d-3485d45988c8","Type":"ContainerDied","Data":"6f8509f4add526bee6f2c530af0bd5aa0a19c70650f80563cf7539e0d4f42a5f"} Oct 30 00:12:08.123559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.123338 4433 scope.go:122] "RemoveContainer" containerID="f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237" Oct 30 00:12:08.132345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.132326 4433 scope.go:122] "RemoveContainer" containerID="f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237" Oct 30 00:12:08.132664 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:12:08.132643 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237\": container with ID starting with f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237 not found: ID does not exist" containerID="f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237" Oct 30 00:12:08.132723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.132674 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237"} err="failed to get container status \"f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237\": rpc error: code = NotFound desc = could not find container \"f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237\": container with ID starting with f87ca113921c058f21067d66069d975f756bdb7af79637c06b151aaa38036237 not found: ID does not exist" Oct 30 00:12:08.177049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.177018 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d789f6877-84tfz"] Oct 30 00:12:08.190460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:08.190428 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d789f6877-84tfz"] Oct 30 00:12:09.177053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:09.177021 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4cbca3-f11b-4658-a33d-3485d45988c8" path="/var/lib/kubelet/pods/4b4cbca3-f11b-4658-a33d-3485d45988c8/volumes" Oct 30 00:12:35.953354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:35.953321 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:12:36.277565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.277492 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-w64bb"] Oct 30 00:12:36.278047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.278030 4433 cpu_manager.go:414] "containerMap: removing container" podUID="4b4cbca3-f11b-4658-a33d-3485d45988c8" containerName="maas-api" Oct 30 00:12:36.278128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.278048 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4cbca3-f11b-4658-a33d-3485d45988c8" containerName="maas-api" Oct 30 00:12:36.278128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.278122 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4cbca3-f11b-4658-a33d-3485d45988c8" containerName="maas-api" Oct 30 00:12:36.281144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.281119 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.281753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.281727 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:12:36.296941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.296915 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 00:12:36.310197 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.310178 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 00:12:36.310319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.310217 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-rnpmz\"" Oct 30 00:12:36.311862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.311840 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-w64bb"] Oct 30 00:12:36.376457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.376424 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8m7\" (UniqueName: \"kubernetes.io/projected/89891569-ea61-4673-b822-25efe5186d3e-kube-api-access-ns8m7\") pod \"ai-gateway-controller-7d85675f98-w64bb\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.376638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.376566 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89891569-ea61-4673-b822-25efe5186d3e-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-w64bb\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.454813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.454780 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:12:36.477846 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.477820 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8m7\" (UniqueName: \"kubernetes.io/projected/89891569-ea61-4673-b822-25efe5186d3e-kube-api-access-ns8m7\") pod \"ai-gateway-controller-7d85675f98-w64bb\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.477956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.477910 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89891569-ea61-4673-b822-25efe5186d3e-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-w64bb\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.480373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.480349 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89891569-ea61-4673-b822-25efe5186d3e-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-w64bb\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.501631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.501607 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8m7\" (UniqueName: \"kubernetes.io/projected/89891569-ea61-4673-b822-25efe5186d3e-kube-api-access-ns8m7\") pod \"ai-gateway-controller-7d85675f98-w64bb\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.592360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.592339 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:36.885105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:36.885054 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-w64bb"] Oct 30 00:12:36.887435 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:12:36.887414 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89891569_ea61_4673_b822_25efe5186d3e.slice/crio-58aaebe00e5a27130dedd4d14ce44cf62dbbc43b378eb9c2501f7c0d5b96ea59 WatchSource:0}: Error finding container 58aaebe00e5a27130dedd4d14ce44cf62dbbc43b378eb9c2501f7c0d5b96ea59: Status 404 returned error can't find the container with id 58aaebe00e5a27130dedd4d14ce44cf62dbbc43b378eb9c2501f7c0d5b96ea59 Oct 30 00:12:37.099828 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:12:37.099780 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:12:37.256144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:37.256061 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" event={"ID":"89891569-ea61-4673-b822-25efe5186d3e","Type":"ContainerStarted","Data":"58aaebe00e5a27130dedd4d14ce44cf62dbbc43b378eb9c2501f7c0d5b96ea59"} Oct 30 00:12:39.266804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:39.266770 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" event={"ID":"89891569-ea61-4673-b822-25efe5186d3e","Type":"ContainerStarted","Data":"f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a"} Oct 30 00:12:39.267277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:39.266897 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:39.309065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:39.309019 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" podStartSLOduration=1.760066255 podStartE2EDuration="3.309006802s" totalImagesPullingTime="1.548940547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:39.308127198 +0000 UTC m=+468.242195738" watchObservedRunningTime="2026-10-30 00:12:39.309006802 +0000 UTC m=+468.243075336" Oct 30 00:12:40.683938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.683899 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:12:40.789229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.789198 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:12:40.856229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.856196 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v"] Oct 30 00:12:40.858986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.858967 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:40.877873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.877826 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 00:12:40.891120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.891100 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-89wls\"" Oct 30 00:12:40.892626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.892607 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v"] Oct 30 00:12:40.944051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.943977 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-sth25"] Oct 30 00:12:40.946517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.946497 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:40.966343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:40.966320 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-sth25"] Oct 30 00:12:41.022531 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.022496 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ed5b12-1185-4013-a1f1-4d98879d0efb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fhm5v\" (UID: \"90ed5b12-1185-4013-a1f1-4d98879d0efb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:41.022726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.022543 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8r4f\" (UniqueName: \"kubernetes.io/projected/90ed5b12-1185-4013-a1f1-4d98879d0efb-kube-api-access-z8r4f\") pod \"payload-pre-processing-7b98bb678c-fhm5v\" (UID: \"90ed5b12-1185-4013-a1f1-4d98879d0efb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:41.123554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.123516 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcdn\" (UniqueName: \"kubernetes.io/projected/4ba6aa75-07db-49e6-b604-d5c0ebd6133c-kube-api-access-rfcdn\") pod \"payload-processing-5f78687d5b-sth25\" (UID: \"4ba6aa75-07db-49e6-b604-d5c0ebd6133c\") " pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:41.123737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.123568 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba6aa75-07db-49e6-b604-d5c0ebd6133c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-sth25\" (UID: \"4ba6aa75-07db-49e6-b604-d5c0ebd6133c\") " pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:41.123737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.123705 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ed5b12-1185-4013-a1f1-4d98879d0efb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fhm5v\" (UID: \"90ed5b12-1185-4013-a1f1-4d98879d0efb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:41.123813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.123755 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8r4f\" (UniqueName: \"kubernetes.io/projected/90ed5b12-1185-4013-a1f1-4d98879d0efb-kube-api-access-z8r4f\") pod \"payload-pre-processing-7b98bb678c-fhm5v\" (UID: \"90ed5b12-1185-4013-a1f1-4d98879d0efb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:41.124398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.124380 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ed5b12-1185-4013-a1f1-4d98879d0efb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fhm5v\" (UID: \"90ed5b12-1185-4013-a1f1-4d98879d0efb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:41.147866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.147838 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8r4f\" (UniqueName: \"kubernetes.io/projected/90ed5b12-1185-4013-a1f1-4d98879d0efb-kube-api-access-z8r4f\") pod \"payload-pre-processing-7b98bb678c-fhm5v\" (UID: \"90ed5b12-1185-4013-a1f1-4d98879d0efb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:41.170414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.170387 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:41.225301 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.225268 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcdn\" (UniqueName: \"kubernetes.io/projected/4ba6aa75-07db-49e6-b604-d5c0ebd6133c-kube-api-access-rfcdn\") pod \"payload-processing-5f78687d5b-sth25\" (UID: \"4ba6aa75-07db-49e6-b604-d5c0ebd6133c\") " pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:41.225463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.225438 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba6aa75-07db-49e6-b604-d5c0ebd6133c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-sth25\" (UID: \"4ba6aa75-07db-49e6-b604-d5c0ebd6133c\") " pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:41.226343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.226319 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba6aa75-07db-49e6-b604-d5c0ebd6133c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-sth25\" (UID: \"4ba6aa75-07db-49e6-b604-d5c0ebd6133c\") " pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:41.250630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.249047 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcdn\" (UniqueName: \"kubernetes.io/projected/4ba6aa75-07db-49e6-b604-d5c0ebd6133c-kube-api-access-rfcdn\") pod \"payload-processing-5f78687d5b-sth25\" (UID: \"4ba6aa75-07db-49e6-b604-d5c0ebd6133c\") " pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:41.258791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.258771 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:41.467087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.467062 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v"] Oct 30 00:12:41.469280 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:12:41.469221 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ed5b12_1185_4013_a1f1_4d98879d0efb.slice/crio-4d65d03b477438e67742681ff9ff78b33cb28fc300f5383371344d5cdcf72780 WatchSource:0}: Error finding container 4d65d03b477438e67742681ff9ff78b33cb28fc300f5383371344d5cdcf72780: Status 404 returned error can't find the container with id 4d65d03b477438e67742681ff9ff78b33cb28fc300f5383371344d5cdcf72780 Oct 30 00:12:41.553777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:41.553756 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-sth25"] Oct 30 00:12:41.556257 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:12:41.556232 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba6aa75_07db_49e6_b604_d5c0ebd6133c.slice/crio-d24e32c6a4699da1ea5732018c864773be51beae2ba774e7caa8cdcfdac35e1d WatchSource:0}: Error finding container d24e32c6a4699da1ea5732018c864773be51beae2ba774e7caa8cdcfdac35e1d: Status 404 returned error can't find the container with id d24e32c6a4699da1ea5732018c864773be51beae2ba774e7caa8cdcfdac35e1d Oct 30 00:12:42.286361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:42.286329 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-sth25" event={"ID":"4ba6aa75-07db-49e6-b604-d5c0ebd6133c","Type":"ContainerStarted","Data":"d24e32c6a4699da1ea5732018c864773be51beae2ba774e7caa8cdcfdac35e1d"} Oct 30 00:12:42.288112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:42.288057 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" event={"ID":"90ed5b12-1185-4013-a1f1-4d98879d0efb","Type":"ContainerStarted","Data":"4d65d03b477438e67742681ff9ff78b33cb28fc300f5383371344d5cdcf72780"} Oct 30 00:12:44.301558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:44.301514 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-sth25" event={"ID":"4ba6aa75-07db-49e6-b604-d5c0ebd6133c","Type":"ContainerStarted","Data":"6f170f9fdc9db75bc39014253d15b02a16df26940aba6cf0a64fced3570f4522"} Oct 30 00:12:44.302022 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:44.301691 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:44.303089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:44.303068 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" event={"ID":"90ed5b12-1185-4013-a1f1-4d98879d0efb","Type":"ContainerStarted","Data":"d35eda50cebcc4e1a073bde0eff30c7db6359552d0114ac726a310a3b5dd64f7"} Oct 30 00:12:44.303199 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:44.303188 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:12:44.343823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:44.343781 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-sth25" podStartSLOduration=2.191050289 podStartE2EDuration="4.343769576s" totalImagesPullingTime="2.152719287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:44.342080067 +0000 UTC m=+473.276148600" watchObservedRunningTime="2026-10-30 00:12:44.343769576 +0000 UTC m=+473.277838111" Oct 30 00:12:44.383644 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:44.383598 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" podStartSLOduration=2.099545388 podStartE2EDuration="4.383582154s" totalImagesPullingTime="2.284036766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:44.38222596 +0000 UTC m=+473.316294505" watchObservedRunningTime="2026-10-30 00:12:44.383582154 +0000 UTC m=+473.317650689" Oct 30 00:12:50.277494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.277459 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:12:50.567480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.567447 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-697599cf87-kg92r"] Oct 30 00:12:50.569983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.569967 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:50.589599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.589555 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697599cf87-kg92r"] Oct 30 00:12:50.605877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.605846 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffe9b07-9588-402f-9578-039cdf337420-metrics-certs\") pod \"ai-gateway-controller-697599cf87-kg92r\" (UID: \"4ffe9b07-9588-402f-9578-039cdf337420\") " pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:50.606004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.605903 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdrcp\" (UniqueName: \"kubernetes.io/projected/4ffe9b07-9588-402f-9578-039cdf337420-kube-api-access-bdrcp\") pod \"ai-gateway-controller-697599cf87-kg92r\" (UID: \"4ffe9b07-9588-402f-9578-039cdf337420\") " pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:50.706976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.706947 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdrcp\" (UniqueName: \"kubernetes.io/projected/4ffe9b07-9588-402f-9578-039cdf337420-kube-api-access-bdrcp\") pod \"ai-gateway-controller-697599cf87-kg92r\" (UID: \"4ffe9b07-9588-402f-9578-039cdf337420\") " pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:50.707112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.707024 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffe9b07-9588-402f-9578-039cdf337420-metrics-certs\") pod \"ai-gateway-controller-697599cf87-kg92r\" (UID: \"4ffe9b07-9588-402f-9578-039cdf337420\") " pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:50.709451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.709420 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffe9b07-9588-402f-9578-039cdf337420-metrics-certs\") pod \"ai-gateway-controller-697599cf87-kg92r\" (UID: \"4ffe9b07-9588-402f-9578-039cdf337420\") " pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:50.730601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.730565 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdrcp\" (UniqueName: \"kubernetes.io/projected/4ffe9b07-9588-402f-9578-039cdf337420-kube-api-access-bdrcp\") pod \"ai-gateway-controller-697599cf87-kg92r\" (UID: \"4ffe9b07-9588-402f-9578-039cdf337420\") " pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:50.881348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:50.881265 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:51.178516 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:12:51.178482 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffe9b07_9588_402f_9578_039cdf337420.slice/crio-070a0261fcb2ee0da8bc7d4d924c00bb304a4471bd66eb3f298d0fe7a04d34ae WatchSource:0}: Error finding container 070a0261fcb2ee0da8bc7d4d924c00bb304a4471bd66eb3f298d0fe7a04d34ae: Status 404 returned error can't find the container with id 070a0261fcb2ee0da8bc7d4d924c00bb304a4471bd66eb3f298d0fe7a04d34ae Oct 30 00:12:51.178753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:51.178734 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697599cf87-kg92r"] Oct 30 00:12:51.336540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:51.336503 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" event={"ID":"4ffe9b07-9588-402f-9578-039cdf337420","Type":"ContainerStarted","Data":"070a0261fcb2ee0da8bc7d4d924c00bb304a4471bd66eb3f298d0fe7a04d34ae"} Oct 30 00:12:52.341987 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:52.341956 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" event={"ID":"4ffe9b07-9588-402f-9578-039cdf337420","Type":"ContainerStarted","Data":"b37a472b51f57a7795ca41afcb7dd30963f13a6342de3c2ae8d2cebac94896cb"} Oct 30 00:12:52.342371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:52.342068 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:12:52.385069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:52.385019 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" podStartSLOduration=2.060089376 podStartE2EDuration="2.38500717s" totalImagesPullingTime="324.917794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:52.384285296 +0000 UTC m=+481.318353855" watchObservedRunningTime="2026-10-30 00:12:52.38500717 +0000 UTC m=+481.319075705" Oct 30 00:12:55.310463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:55.310434 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-sth25" Oct 30 00:12:55.311048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:12:55.310509 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fhm5v" Oct 30 00:13:03.349126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.349091 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-697599cf87-kg92r" Oct 30 00:13:03.417899 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.417869 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-w64bb"] Oct 30 00:13:03.418108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.418084 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" podUID="89891569-ea61-4673-b822-25efe5186d3e" containerName="manager" containerID="cri-o://f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a" gracePeriod=30 Oct 30 00:13:03.661522 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.661497 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:13:03.725614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.725536 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89891569-ea61-4673-b822-25efe5186d3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89891569-ea61-4673-b822-25efe5186d3e-metrics-certs\") pod \"89891569-ea61-4673-b822-25efe5186d3e\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " Oct 30 00:13:03.725751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.725634 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89891569-ea61-4673-b822-25efe5186d3e-kube-api-access-ns8m7\" (UniqueName: \"kubernetes.io/projected/89891569-ea61-4673-b822-25efe5186d3e-kube-api-access-ns8m7\") pod \"89891569-ea61-4673-b822-25efe5186d3e\" (UID: \"89891569-ea61-4673-b822-25efe5186d3e\") " Oct 30 00:13:03.727491 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.727464 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89891569-ea61-4673-b822-25efe5186d3e-metrics-certs" pod "89891569-ea61-4673-b822-25efe5186d3e" (UID: "89891569-ea61-4673-b822-25efe5186d3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:13:03.727626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.727508 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89891569-ea61-4673-b822-25efe5186d3e-kube-api-access-ns8m7" pod "89891569-ea61-4673-b822-25efe5186d3e" (UID: "89891569-ea61-4673-b822-25efe5186d3e"). InnerVolumeSpecName "kube-api-access-ns8m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:13:03.826961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.826926 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns8m7\" (UniqueName: \"kubernetes.io/projected/89891569-ea61-4673-b822-25efe5186d3e-kube-api-access-ns8m7\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:13:03.826961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:03.826957 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89891569-ea61-4673-b822-25efe5186d3e-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:13:04.396051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.396017 4433 generic.go:356] "Generic (PLEG): container finished" podID="89891569-ea61-4673-b822-25efe5186d3e" containerID="f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a" exitCode=0 Oct 30 00:13:04.396517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.396059 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" Oct 30 00:13:04.396517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.396109 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" event={"ID":"89891569-ea61-4673-b822-25efe5186d3e","Type":"ContainerDied","Data":"f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a"} Oct 30 00:13:04.396517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.396172 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-w64bb" event={"ID":"89891569-ea61-4673-b822-25efe5186d3e","Type":"ContainerDied","Data":"58aaebe00e5a27130dedd4d14ce44cf62dbbc43b378eb9c2501f7c0d5b96ea59"} Oct 30 00:13:04.396517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.396205 4433 scope.go:122] "RemoveContainer" containerID="f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a" Oct 30 00:13:04.404947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.404929 4433 scope.go:122] "RemoveContainer" containerID="f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a" Oct 30 00:13:04.405174 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:13:04.405157 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a\": container with ID starting with f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a not found: ID does not exist" containerID="f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a" Oct 30 00:13:04.405228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.405184 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a"} err="failed to get container status \"f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a\": rpc error: code = NotFound desc = could not find container \"f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a\": container with ID starting with f27027a5c583926ad670bfcb0271ecbebcd70a8e520ce7cf4ed46a3ba829684a not found: ID does not exist" Oct 30 00:13:04.447902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.447877 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-w64bb"] Oct 30 00:13:04.460951 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:04.460929 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-w64bb"] Oct 30 00:13:05.130041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:05.129992 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:05.178071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:05.178041 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89891569-ea61-4673-b822-25efe5186d3e" path="/var/lib/kubelet/pods/89891569-ea61-4673-b822-25efe5186d3e/volumes" Oct 30 00:13:05.651998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:05.651962 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:06.943272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:06.943237 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:07.063187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:07.063134 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:07.098724 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:13:07.098694 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:13:12.244302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.244269 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4"] Oct 30 00:13:12.244888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.244872 4433 cpu_manager.go:414] "containerMap: removing container" podUID="89891569-ea61-4673-b822-25efe5186d3e" containerName="manager" Oct 30 00:13:12.244934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.244891 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89891569-ea61-4673-b822-25efe5186d3e" containerName="manager" Oct 30 00:13:12.245040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.245030 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="89891569-ea61-4673-b822-25efe5186d3e" containerName="manager" Oct 30 00:13:12.248373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.248353 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.264906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.264882 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 00:13:12.277724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.277704 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 00:13:12.277928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.277910 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 00:13:12.277988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.277917 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vrftc\"" Oct 30 00:13:12.279812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.279793 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4"] Oct 30 00:13:12.304138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.304088 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:12.404558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.404522 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.404755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.404657 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzrb\" (UniqueName: \"kubernetes.io/projected/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-kube-api-access-czzrb\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.404755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.404715 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.404834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.404755 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.404886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.404865 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.404965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.404949 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.505696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.505607 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.505696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.505673 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.505923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.505717 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzrb\" (UniqueName: \"kubernetes.io/projected/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-kube-api-access-czzrb\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.505923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.505759 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.505923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.505797 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.506100 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.505940 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.506165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.506140 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.506214 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.506177 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.506316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.506295 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.507852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.507833 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.508115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.508098 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.529583 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.529541 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzrb\" (UniqueName: \"kubernetes.io/projected/12e5f30f-344a-4a0c-82ac-60ceb5fbebf2-kube-api-access-czzrb\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4\" (UID: \"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.560216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.560177 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:12.861091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:12.861067 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4"] Oct 30 00:13:12.863256 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:13:12.863232 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e5f30f_344a_4a0c_82ac_60ceb5fbebf2.slice/crio-cf481f6f9a0706002f3f7a411cf3e8f104d82c868d4f317580e492adce665e3b WatchSource:0}: Error finding container cf481f6f9a0706002f3f7a411cf3e8f104d82c868d4f317580e492adce665e3b: Status 404 returned error can't find the container with id cf481f6f9a0706002f3f7a411cf3e8f104d82c868d4f317580e492adce665e3b Oct 30 00:13:13.438556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:13.438516 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" event={"ID":"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2","Type":"ContainerStarted","Data":"cf481f6f9a0706002f3f7a411cf3e8f104d82c868d4f317580e492adce665e3b"} Oct 30 00:13:14.444632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:14.444589 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" event={"ID":"12e5f30f-344a-4a0c-82ac-60ceb5fbebf2","Type":"ContainerStarted","Data":"e5021d49ac80a3821ce77bbc92c9bd84d02054d74ac87b30a80de72aae3f46c5"} Oct 30 00:13:14.488286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:14.488230 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" podStartSLOduration=1.382464557 podStartE2EDuration="2.488216165s" totalImagesPullingTime="1.105751608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:14.487836656 +0000 UTC m=+503.421905192" watchObservedRunningTime="2026-10-30 00:13:14.488216165 +0000 UTC m=+503.422284701" Oct 30 00:13:14.560756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:14.560725 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:16.428833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.428802 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln"] Oct 30 00:13:16.433979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.433956 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.456679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.456650 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln"] Oct 30 00:13:16.465106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.465085 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 00:13:16.482079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.482043 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:16.541532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.541503 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6zv\" (UniqueName: \"kubernetes.io/projected/f6aae325-01b5-4ab6-8360-c0e67efd8e43-kube-api-access-zd6zv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.541692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.541569 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aae325-01b5-4ab6-8360-c0e67efd8e43-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.541762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.541743 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.541799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.541770 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.541865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.541852 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.541908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.541890 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643379 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aae325-01b5-4ab6-8360-c0e67efd8e43-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643423 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643440 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643460 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643587 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643679 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6zv\" (UniqueName: \"kubernetes.io/projected/f6aae325-01b5-4ab6-8360-c0e67efd8e43-kube-api-access-zd6zv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643861 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.643959 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.643936 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.644045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.644022 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.645657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.645640 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aae325-01b5-4ab6-8360-c0e67efd8e43-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.646037 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.646017 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aae325-01b5-4ab6-8360-c0e67efd8e43-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.667714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.667683 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6zv\" (UniqueName: \"kubernetes.io/projected/f6aae325-01b5-4ab6-8360-c0e67efd8e43-kube-api-access-zd6zv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bw4ln\" (UID: \"f6aae325-01b5-4ab6-8360-c0e67efd8e43\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:16.746639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:16.746546 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:17.045754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:17.045730 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln"] Oct 30 00:13:17.048897 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:13:17.048864 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aae325_01b5_4ab6_8360_c0e67efd8e43.slice/crio-cf5f4bba3bc2b131f6d169eb308b599af0af2b6fe2400da3b3758db59f8a166f WatchSource:0}: Error finding container cf5f4bba3bc2b131f6d169eb308b599af0af2b6fe2400da3b3758db59f8a166f: Status 404 returned error can't find the container with id cf5f4bba3bc2b131f6d169eb308b599af0af2b6fe2400da3b3758db59f8a166f Oct 30 00:13:17.459480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:17.459448 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" event={"ID":"f6aae325-01b5-4ab6-8360-c0e67efd8e43","Type":"ContainerStarted","Data":"f5f5a916c0bb3caa764e8e2bdfc79a064f78971c6e69a08b91a72099999be622"} Oct 30 00:13:17.459895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:17.459497 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" event={"ID":"f6aae325-01b5-4ab6-8360-c0e67efd8e43","Type":"ContainerStarted","Data":"cf5f4bba3bc2b131f6d169eb308b599af0af2b6fe2400da3b3758db59f8a166f"} Oct 30 00:13:17.503132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:17.503083 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" podStartSLOduration=1.3214455059999999 podStartE2EDuration="1.503069551s" totalImagesPullingTime="181.624045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:17.502629029 +0000 UTC m=+506.436697561" watchObservedRunningTime="2026-10-30 00:13:17.503069551 +0000 UTC m=+506.437138085" Oct 30 00:13:17.747701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:17.747623 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:19.854544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.854511 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8"] Oct 30 00:13:19.857062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.857045 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:19.872706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.872680 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 00:13:19.887739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.887712 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8"] Oct 30 00:13:19.907545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.907515 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:19.972041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.972009 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:19.972041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.972039 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:19.972230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.972075 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bb8\" (UniqueName: \"kubernetes.io/projected/6b2916e1-b456-4941-a17e-235744eac859-kube-api-access-82bb8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:19.972230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.972181 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:19.972230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.972219 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:19.972341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:19.972249 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2916e1-b456-4941-a17e-235744eac859-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.072782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.072745 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.072782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.072783 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.072991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.072913 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bb8\" (UniqueName: \"kubernetes.io/projected/6b2916e1-b456-4941-a17e-235744eac859-kube-api-access-82bb8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.072991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.072953 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.073063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.073000 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.073063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.073028 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2916e1-b456-4941-a17e-235744eac859-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.073159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.073138 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.073209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.073180 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.073327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.073308 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.075254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.075228 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2916e1-b456-4941-a17e-235744eac859-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.075450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.075435 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2916e1-b456-4941-a17e-235744eac859-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.096955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.096922 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bb8\" (UniqueName: \"kubernetes.io/projected/6b2916e1-b456-4941-a17e-235744eac859-kube-api-access-82bb8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6z2n8\" (UID: \"6b2916e1-b456-4941-a17e-235744eac859\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.170154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.170086 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:20.465165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.465138 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8"] Oct 30 00:13:20.467038 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:13:20.467013 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2916e1_b456_4941_a17e_235744eac859.slice/crio-a6b2519cf2be3b752534cdff3046febc5d2ce23d44db5b0eadbee91652eeb6d4 WatchSource:0}: Error finding container a6b2519cf2be3b752534cdff3046febc5d2ce23d44db5b0eadbee91652eeb6d4: Status 404 returned error can't find the container with id a6b2519cf2be3b752534cdff3046febc5d2ce23d44db5b0eadbee91652eeb6d4 Oct 30 00:13:20.473128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:20.473104 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" event={"ID":"6b2916e1-b456-4941-a17e-235744eac859","Type":"ContainerStarted","Data":"a6b2519cf2be3b752534cdff3046febc5d2ce23d44db5b0eadbee91652eeb6d4"} Oct 30 00:13:21.121197 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:21.121165 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:21.479421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:21.479328 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" event={"ID":"6b2916e1-b456-4941-a17e-235744eac859","Type":"ContainerStarted","Data":"7b7c6322ee7ad23f2c07cd034e35392bab202b7846b0e0ec2afa77ffcb2df90c"} Oct 30 00:13:21.529995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:21.529939 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" podStartSLOduration=2.354927117 podStartE2EDuration="2.529925267s" totalImagesPullingTime="174.99815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:21.527820902 +0000 UTC m=+510.461889436" watchObservedRunningTime="2026-10-30 00:13:21.529925267 +0000 UTC m=+510.463993802" Oct 30 00:13:22.170249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.170206 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:22.471679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.471599 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k"] Oct 30 00:13:22.474231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.474211 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.493002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.492947 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 00:13:22.494514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.494487 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k"] Oct 30 00:13:22.521541 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.521506 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:22.560476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.560442 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:22.573514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.573482 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:22.595042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.595006 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.595179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.595066 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.595179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.595162 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.595272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.595245 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.595351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.595337 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc492\" (UniqueName: \"kubernetes.io/projected/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-kube-api-access-qc492\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.595391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.595384 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696118 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696175 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696362 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696460 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc492\" (UniqueName: \"kubernetes.io/projected/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-kube-api-access-qc492\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696541 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696528 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696613 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696621 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696649 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.696856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.696834 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.698896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.698873 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.699066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.699049 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.721152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.721112 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc492\" (UniqueName: \"kubernetes.io/projected/8e26ca3e-0c37-482b-81e5-ac51211ad7b6-kube-api-access-qc492\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qs69k\" (UID: \"8e26ca3e-0c37-482b-81e5-ac51211ad7b6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:22.788322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:22.788222 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:23.085706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:23.085680 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k"] Oct 30 00:13:23.087824 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:13:23.087792 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e26ca3e_0c37_482b_81e5_ac51211ad7b6.slice/crio-353d373e27f756db8119a66231ef9acf7362d6b2c1a272479a28b906a007619b WatchSource:0}: Error finding container 353d373e27f756db8119a66231ef9acf7362d6b2c1a272479a28b906a007619b: Status 404 returned error can't find the container with id 353d373e27f756db8119a66231ef9acf7362d6b2c1a272479a28b906a007619b Oct 30 00:13:23.490717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:23.490623 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" event={"ID":"8e26ca3e-0c37-482b-81e5-ac51211ad7b6","Type":"ContainerStarted","Data":"ff496e31393a060ac9641f3659442c6f735e173fae5885dc5e86520fa413e532"} Oct 30 00:13:23.490717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:23.490682 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" event={"ID":"8e26ca3e-0c37-482b-81e5-ac51211ad7b6","Type":"ContainerStarted","Data":"353d373e27f756db8119a66231ef9acf7362d6b2c1a272479a28b906a007619b"} Oct 30 00:13:23.502484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:23.502458 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4" Oct 30 00:13:23.534123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:23.534071 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" podStartSLOduration=1.345371158 podStartE2EDuration="1.534056656s" totalImagesPullingTime="188.685498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:23.533362033 +0000 UTC m=+512.467430568" watchObservedRunningTime="2026-10-30 00:13:23.534056656 +0000 UTC m=+512.468125191" Oct 30 00:13:23.789034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:23.788932 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:25.859878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:25.859842 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:26.746810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:26.746774 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:26.759294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:26.759270 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:27.521372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.521341 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bw4ln" Oct 30 00:13:27.682278 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.682248 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-spdzp"] Oct 30 00:13:27.684837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.684819 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.705047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.705020 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-spdzp"] Oct 30 00:13:27.709183 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.709158 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 00:13:27.749505 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.749475 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059340ab-5d93-4759-8d08-b5203c258e05-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.749649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.749519 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.749649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.749623 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.749713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.749646 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-home\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.749787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.749731 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.749787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.749774 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmd4\" (UniqueName: \"kubernetes.io/projected/059340ab-5d93-4759-8d08-b5203c258e05-kube-api-access-6jmd4\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.791489 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.791412 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:27.851039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851001 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmd4\" (UniqueName: \"kubernetes.io/projected/059340ab-5d93-4759-8d08-b5203c258e05-kube-api-access-6jmd4\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851076 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059340ab-5d93-4759-8d08-b5203c258e05-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851108 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851132 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851147 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-home\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851174 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851618 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851613 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.851783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.851665 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-home\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.853220 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.853195 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059340ab-5d93-4759-8d08-b5203c258e05-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.853392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.853376 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059340ab-5d93-4759-8d08-b5203c258e05-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.874951 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.874928 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmd4\" (UniqueName: \"kubernetes.io/projected/059340ab-5d93-4759-8d08-b5203c258e05-kube-api-access-6jmd4\") pod \"echo-simulator-kserve-64b6f8f96c-spdzp\" (UID: \"059340ab-5d93-4759-8d08-b5203c258e05\") " pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:27.997586 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:27.997544 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:28.294942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:28.294913 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-spdzp"] Oct 30 00:13:28.296613 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:13:28.296586 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059340ab_5d93_4759_8d08_b5203c258e05.slice/crio-8016074d4db338bdb67b6dd52e1a660d1a114f9d30fb4c26cefb0d700d2c284f WatchSource:0}: Error finding container 8016074d4db338bdb67b6dd52e1a660d1a114f9d30fb4c26cefb0d700d2c284f: Status 404 returned error can't find the container with id 8016074d4db338bdb67b6dd52e1a660d1a114f9d30fb4c26cefb0d700d2c284f Oct 30 00:13:28.528112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:28.527993 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" event={"ID":"059340ab-5d93-4759-8d08-b5203c258e05","Type":"ContainerStarted","Data":"8016074d4db338bdb67b6dd52e1a660d1a114f9d30fb4c26cefb0d700d2c284f"} Oct 30 00:13:29.466162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:29.466123 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:30.171060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.171024 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:30.183605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.183559 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:30.539653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.539534 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" event={"ID":"059340ab-5d93-4759-8d08-b5203c258e05","Type":"ContainerStarted","Data":"3890215dc8195a738de93d89ef3a14641a0e6ed354ed79661effabf2c2dd42ee"} Oct 30 00:13:30.550672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.550646 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6z2n8" Oct 30 00:13:30.584181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.584131 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" podStartSLOduration=2.228721595 podStartE2EDuration="3.584118747s" totalImagesPullingTime="1.355397152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:30.582497625 +0000 UTC m=+519.516566174" watchObservedRunningTime="2026-10-30 00:13:30.584118747 +0000 UTC m=+519.518187281" Oct 30 00:13:30.755410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.755374 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v"] Oct 30 00:13:30.757942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.757924 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.777040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.777009 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 00:13:30.779630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.779601 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.779780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.779664 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.779780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.779696 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.779896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.779791 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbsn\" (UniqueName: \"kubernetes.io/projected/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-kube-api-access-vnbsn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.779896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.779822 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.779896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.779849 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.780071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.779943 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v"] Oct 30 00:13:30.825944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.825905 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:30.881129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881089 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbsn\" (UniqueName: \"kubernetes.io/projected/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-kube-api-access-vnbsn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881135 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881165 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881211 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881249 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881275 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881653 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881716 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.881929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.881875 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.883522 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.883500 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.883792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.883774 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.905091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.905060 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbsn\" (UniqueName: \"kubernetes.io/projected/de398e8d-5cd0-4e2f-ae60-7bf17a4a942e-kube-api-access-vnbsn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v\" (UID: \"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:30.998168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:30.998132 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:31.087203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:31.087122 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:31.386024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:31.386000 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v"] Oct 30 00:13:31.387755 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:13:31.387730 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde398e8d_5cd0_4e2f_ae60_7bf17a4a942e.slice/crio-48824239b2c8a71b392c740a9de1979b1f9d0c4792db7d2755d7b609e8d5fb66 WatchSource:0}: Error finding container 48824239b2c8a71b392c740a9de1979b1f9d0c4792db7d2755d7b609e8d5fb66: Status 404 returned error can't find the container with id 48824239b2c8a71b392c740a9de1979b1f9d0c4792db7d2755d7b609e8d5fb66 Oct 30 00:13:31.546018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:31.545978 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" event={"ID":"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e","Type":"ContainerStarted","Data":"48824239b2c8a71b392c740a9de1979b1f9d0c4792db7d2755d7b609e8d5fb66"} Oct 30 00:13:32.552229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:32.552190 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" event={"ID":"de398e8d-5cd0-4e2f-ae60-7bf17a4a942e","Type":"ContainerStarted","Data":"61773a746346493da3766e6312d459cc2bc26f4ac6c3daa4f17b6c962030c312"} Oct 30 00:13:32.599087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:32.599035 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podStartSLOduration=2.433800447 podStartE2EDuration="2.599020803s" totalImagesPullingTime="165.220356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:32.59804674 +0000 UTC m=+521.532115298" watchObservedRunningTime="2026-10-30 00:13:32.599020803 +0000 UTC m=+521.533089338" Oct 30 00:13:32.789370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:32.789332 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:32.802297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:32.802230 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:33.088168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:33.088133 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:33.568554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:33.568528 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qs69k" Oct 30 00:13:34.919140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:34.919096 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj"] Oct 30 00:13:34.922046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:34.922022 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:34.932868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:34.932841 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z"] Oct 30 00:13:34.935783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:34.935759 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:34.948006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:34.947976 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj"] Oct 30 00:13:34.951514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:34.951479 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z"] Oct 30 00:13:35.023594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023538 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.023793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023609 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.023793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023654 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d990bb4a-b08e-49de-ba43-2181a0996cea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.023793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023769 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.023963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023808 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.023963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023858 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.023963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023950 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.024122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.023972 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.024122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024000 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.024122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024002 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:13:35.024122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024076 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tgqt4\"" Oct 30 00:13:35.024122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024081 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.024122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024122 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.024397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024145 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.024397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024196 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.024397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024233 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42dt\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-kube-api-access-q42dt\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.024397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024336 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.024397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024370 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.024627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024413 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.024627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.024446 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgqp\" (UniqueName: \"kubernetes.io/projected/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-kube-api-access-4jgqp\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.125159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125124 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.125159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125161 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.125395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125181 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.125395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125307 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.125395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125354 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.125395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125384 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.125649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125399 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.125649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125428 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42dt\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-kube-api-access-q42dt\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.125649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125481 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.125649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125503 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.125649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125510 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.125649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125595 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.125649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125627 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgqp\" (UniqueName: \"kubernetes.io/projected/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-kube-api-access-4jgqp\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125673 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125712 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125737 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d990bb4a-b08e-49de-ba43-2181a0996cea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125752 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125770 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125815 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125845 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.126000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.125840 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.126177 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.126187 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.126447 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.126507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.126447 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.126644 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.126508 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d990bb4a-b08e-49de-ba43-2181a0996cea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.126729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.126699 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.126993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.126967 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.128271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.128246 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.128271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.128266 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.128393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.128268 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.128393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.128314 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.148929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.148897 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.149068 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.149018 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.149227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.149208 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgqp\" (UniqueName: \"kubernetes.io/projected/e0d5fcfc-fadc-44e6-9b61-986bc7395aca-kube-api-access-4jgqp\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cj97z\" (UID: \"e0d5fcfc-fadc-44e6-9b61-986bc7395aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.149284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.149236 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42dt\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-kube-api-access-q42dt\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-phppj\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.235523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.235444 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:35.252510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.252478 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:35.566664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.566502 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj"] Oct 30 00:13:35.569884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:35.569862 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z"] Oct 30 00:13:36.078793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:36.078756 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:36.577136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:36.577091 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" event={"ID":"e0d5fcfc-fadc-44e6-9b61-986bc7395aca","Type":"ContainerStarted","Data":"66e8fcc51bad37d7aa19d81b41438b275b41cc3b5c666fd7793f9e7bc958ce2d"} Oct 30 00:13:36.578746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:36.578717 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" event={"ID":"d990bb4a-b08e-49de-ba43-2181a0996cea","Type":"ContainerStarted","Data":"4f9bfe064326f06d0fa320bddd22c68922d716f759effd7475eb0a881d6b9fc6"} Oct 30 00:13:37.109737 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:13:37.109699 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:13:37.998683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:37.998645 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:38.014896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:38.014861 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:38.377285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:38.377251 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:38.377612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:38.377320 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:38.377612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:38.377351 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:38.592931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:38.592889 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" event={"ID":"d990bb4a-b08e-49de-ba43-2181a0996cea","Type":"ContainerStarted","Data":"d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d"} Oct 30 00:13:38.605927 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:38.605897 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-spdzp" Oct 30 00:13:38.641823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:38.641695 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podStartSLOduration=1.8445845680000001 podStartE2EDuration="4.641673581s" totalImagesPullingTime="2.797089013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:38.639786943 +0000 UTC m=+527.573855498" watchObservedRunningTime="2026-10-30 00:13:38.641673581 +0000 UTC m=+527.575742119" Oct 30 00:13:39.235696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:39.235654 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:39.237197 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:39.237174 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:39.237272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:39.237226 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:39.607728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:39.607690 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:39.608061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:39.607761 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:39.608061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:39.607790 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:40.236591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:40.236534 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:40.236746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:40.236624 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:40.611927 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:40.611883 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" event={"ID":"e0d5fcfc-fadc-44e6-9b61-986bc7395aca","Type":"ContainerStarted","Data":"65e41017343c27d5711a1f90b11827fc32c970507fa17d54bbfcdfca9ec56545"} Oct 30 00:13:41.100729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.100685 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:41.114395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.114364 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:41.172598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.171566 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podStartSLOduration=3.144081451 podStartE2EDuration="7.171544604s" totalImagesPullingTime="4.027463153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:40.686019173 +0000 UTC m=+529.620087714" watchObservedRunningTime="2026-10-30 00:13:41.171544604 +0000 UTC m=+530.105613140" Oct 30 00:13:41.235963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.235924 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:41.236126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.235987 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:41.274598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.274186 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:41.280694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.280662 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:13:41.280928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.280901 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:13:41.627904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:41.627871 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:13:42.235824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:42.235785 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:42.236011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:42.235837 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:42.273597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:42.266673 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:13:42.273597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:42.266729 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:13:43.235984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:43.235952 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:43.236368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:43.236001 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:43.278156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:43.277692 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:13:43.278156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:43.277753 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:13:44.236039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:44.235986 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:44.236627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:44.236063 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:44.256007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:44.255968 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:13:44.256256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:44.256224 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:13:45.235863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:45.235823 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:45.236034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:45.235890 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:45.262598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:45.260038 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:13:45.262598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:45.260103 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:13:46.236925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:46.236876 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:13:46.237101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:46.236966 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:13:46.253176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:46.253147 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:13:46.253253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:46.253211 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:13:47.253417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:47.253376 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:13:47.253839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:47.253446 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:13:47.285932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:47.285895 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:47.286774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:47.286741 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:47.287632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:47.287612 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:13:48.329932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:48.329895 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:48.330791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:48.330762 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:48.331521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:48.331499 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" Oct 30 00:13:49.962560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.962522 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn"] Oct 30 00:13:49.966829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.966806 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.992432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.992406 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn"] Oct 30 00:13:49.998842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.998813 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.998944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.998851 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.998993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.998951 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jns6\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-kube-api-access-4jns6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.999028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.999016 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.999065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.999036 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.999065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.999058 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.999158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.999140 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.999263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.999246 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:49.999326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:49.999297 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306325e3-eb7b-4199-93f1-27016cd6e3ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100236 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100193 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100271 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100309 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306325e3-eb7b-4199-93f1-27016cd6e3ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100335 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100364 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100436 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jns6\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-kube-api-access-4jns6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100517 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100545 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100609 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100749 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100758 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.100862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100814 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.101013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.100903 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.101101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.101082 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306325e3-eb7b-4199-93f1-27016cd6e3ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.102868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.102843 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.103044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.103026 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.125320 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.125288 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.125415 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.125343 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jns6\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-kube-api-access-4jns6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-48vhn\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.280683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.280589 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:50.594985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.594956 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn"] Oct 30 00:13:50.596561 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:13:50.596538 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306325e3_eb7b_4199_93f1_27016cd6e3ff.slice/crio-3aec28174c253eacdf9ebcce0c3de254a7725ab4220bd0dd29a4221d62517f99 WatchSource:0}: Error finding container 3aec28174c253eacdf9ebcce0c3de254a7725ab4220bd0dd29a4221d62517f99: Status 404 returned error can't find the container with id 3aec28174c253eacdf9ebcce0c3de254a7725ab4220bd0dd29a4221d62517f99 Oct 30 00:13:50.600547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.600515 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:50.600677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.600609 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:50.600677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.600658 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:13:50.666749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.666725 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" event={"ID":"306325e3-eb7b-4199-93f1-27016cd6e3ff","Type":"ContainerStarted","Data":"5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de"} Oct 30 00:13:50.666858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.666778 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" event={"ID":"306325e3-eb7b-4199-93f1-27016cd6e3ff","Type":"ContainerStarted","Data":"3aec28174c253eacdf9ebcce0c3de254a7725ab4220bd0dd29a4221d62517f99"} Oct 30 00:13:50.716783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.716733 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podStartSLOduration=1.716718443 podStartE2EDuration="1.716718443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:50.714480616 +0000 UTC m=+539.648549170" watchObservedRunningTime="2026-10-30 00:13:50.716718443 +0000 UTC m=+539.650786977" Oct 30 00:13:50.780477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.780437 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:50.957192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:50.957108 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:51.281433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:51.281398 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:51.283156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:51.283131 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:51.283276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:51.283180 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:52.281080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:52.281043 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:52.281262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:52.281100 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:52.478500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:52.478464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:52.595253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:52.595215 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:13:53.281100 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:53.281063 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:53.281282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:53.281118 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:54.281088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:54.281047 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:54.281488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:54.281102 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:55.281751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:55.281714 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:55.282150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:55.281765 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:56.281424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:56.281385 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:56.281629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:56.281444 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:57.281490 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:57.281460 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:57.281875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:57.281507 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:58.281272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:58.281226 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-48vhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:13:58.281440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:58.281296 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 00:13:59.285450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:59.285421 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:59.285916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:59.285823 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:13:59.286462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:13:59.286443 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:14:07.097169 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:14:07.097131 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:14:27.831109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:27.831070 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.019798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.019763 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.130427 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.130332 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.256075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.256037 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.376268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.376238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.490598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.490484 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.600106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.600074 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.696162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.696132 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8"] Oct 30 00:14:28.699234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.699209 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.719141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.719114 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-185595da-openshift-default-dockercfg-vchnj\"" Oct 30 00:14:28.734369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.734345 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8"] Oct 30 00:14:28.807724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.807698 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-data\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.807864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.807751 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-socket\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.807948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.807860 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-credential-socket\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.807948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.807896 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-token\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.807948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.807945 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9gq\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-kube-api-access-zp9gq\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.808043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.807973 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-certs\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.808043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.807988 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-podinfo\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.808108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.808059 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f048d9a1-3b15-46b9-8309-e14d0025f24c-istiod-ca-cert\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.808108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.808077 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-envoy\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.909706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909659 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-socket\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909740 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-credential-socket\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909772 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-token\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909811 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9gq\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-kube-api-access-zp9gq\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909849 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-certs\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909873 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-podinfo\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909910 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f048d9a1-3b15-46b9-8309-e14d0025f24c-istiod-ca-cert\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909932 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-envoy\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.909992 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-data\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.910119 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-socket\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.910328 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-data\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.910412 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-certs\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.910931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.910807 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-credential-socket\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.911039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.911019 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f048d9a1-3b15-46b9-8309-e14d0025f24c-istiod-ca-cert\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.912686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.912660 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-envoy\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.912926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.912899 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-podinfo\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.928989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.928965 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:28.934657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.934638 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-token\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:28.934869 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:28.934853 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9gq\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-kube-api-access-zp9gq\") pod \"e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:29.015127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.015083 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:29.063947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.063847 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:29.137249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.137220 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8"] Oct 30 00:14:29.141216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.141195 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.159904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.159857 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6c9e9fa7-openshift-default-dockercfg-zd9l6\"" Oct 30 00:14:29.162599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.162557 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8"] Oct 30 00:14:29.186309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.186281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:29.213349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213317 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-token\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213352 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istiod-ca-cert\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213590 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213515 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-data\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213599 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-credential-socket\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213638 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-envoy\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213690 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-certs\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213749 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ltl\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-kube-api-access-h9ltl\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213782 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-podinfo\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.213883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.213816 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-socket\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.303393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.303356 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:29.315191 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315103 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istiod-ca-cert\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.315353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315279 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-data\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315426 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-credential-socket\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315456 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-envoy\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315514 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-certs\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315567 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ltl\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-kube-api-access-h9ltl\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315646 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-podinfo\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315662 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-data\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315894 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istiod-ca-cert\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315931 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-socket\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.315968 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-credential-socket\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.316127 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-certs\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.316204 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-socket\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.316755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.316369 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-token\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.318212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.318190 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-envoy\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.318357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.318339 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-podinfo\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.336340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.336324 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8"] Oct 30 00:14:29.338544 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:29.338522 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf048d9a1_3b15_46b9_8309_e14d0025f24c.slice/crio-1a81b17e4b276a12e73a2ea59cbb398c2fff0d286795d77b59203abe556a395b WatchSource:0}: Error finding container 1a81b17e4b276a12e73a2ea59cbb398c2fff0d286795d77b59203abe556a395b: Status 404 returned error can't find the container with id 1a81b17e4b276a12e73a2ea59cbb398c2fff0d286795d77b59203abe556a395b Oct 30 00:14:29.340974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.340943 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:29.341045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.341012 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:29.341045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.341041 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:29.345235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.345217 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-token\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.345311 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.345250 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ltl\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-kube-api-access-h9ltl\") pod \"e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.454335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.454307 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:29.474657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.474600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:29.583651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.583521 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:29.778409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.778385 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8"] Oct 30 00:14:29.780009 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:29.779981 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1bcfc7_851d_48c7_bc90_3e8e69a36e2b.slice/crio-1f988daa5db3c2c5cd1f44fb1142dadab9c29db02fad57e752e0e71462ba2fcc WatchSource:0}: Error finding container 1f988daa5db3c2c5cd1f44fb1142dadab9c29db02fad57e752e0e71462ba2fcc: Status 404 returned error can't find the container with id 1f988daa5db3c2c5cd1f44fb1142dadab9c29db02fad57e752e0e71462ba2fcc Oct 30 00:14:29.782367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.782336 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:29.782451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.782397 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:29.782451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.782425 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:29.868702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.868662 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" event={"ID":"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b","Type":"ContainerStarted","Data":"80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c"} Oct 30 00:14:29.868868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.868735 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" event={"ID":"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b","Type":"ContainerStarted","Data":"1f988daa5db3c2c5cd1f44fb1142dadab9c29db02fad57e752e0e71462ba2fcc"} Oct 30 00:14:29.870615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.870583 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" event={"ID":"f048d9a1-3b15-46b9-8309-e14d0025f24c","Type":"ContainerStarted","Data":"d36b6c07b785fecd1bd10ee8c412451e614b50ce8f7b2e227f11523e32fa5066"} Oct 30 00:14:29.870760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.870637 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" event={"ID":"f048d9a1-3b15-46b9-8309-e14d0025f24c","Type":"ContainerStarted","Data":"1a81b17e4b276a12e73a2ea59cbb398c2fff0d286795d77b59203abe556a395b"} Oct 30 00:14:29.950539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.950487 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" podStartSLOduration=0.950473248 podStartE2EDuration="950.473248ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:29.949114402 +0000 UTC m=+578.883182949" watchObservedRunningTime="2026-10-30 00:14:29.950473248 +0000 UTC m=+578.884541782" Oct 30 00:14:29.997861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:29.997806 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podStartSLOduration=1.997794166 podStartE2EDuration="1.997794166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:29.996705451 +0000 UTC m=+578.930774009" watchObservedRunningTime="2026-10-30 00:14:29.997794166 +0000 UTC m=+578.931862698" Oct 30 00:14:30.015731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.015695 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:30.020736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.020711 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:30.210813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.210721 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:30.454489 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.454456 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:30.459321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.459298 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:30.628262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.628224 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs"] Oct 30 00:14:30.632093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.632068 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.651008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.650980 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7505bd-openshift-default-dockercfg-98hdd\"" Oct 30 00:14:30.654595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.654556 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs"] Oct 30 00:14:30.685036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.685007 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:30.736120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736093 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736166 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736283 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736343 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzgm\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-kube-api-access-drzgm\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736423 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736386 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736477 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736531 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509d7476-b7d1-4657-b779-a2876047d78c-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736660 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.736750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.736716 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509d7476-b7d1-4657-b779-a2876047d78c-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.837883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.837849 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.837900 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509d7476-b7d1-4657-b779-a2876047d78c-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838147 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838076 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838168 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838210 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838251 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzgm\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-kube-api-access-drzgm\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838296 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838350 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838384 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838407 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509d7476-b7d1-4657-b779-a2876047d78c-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838675 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838650 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838666 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838707 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509d7476-b7d1-4657-b779-a2876047d78c-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.838875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.838846 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.840985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.840966 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509d7476-b7d1-4657-b779-a2876047d78c-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.844179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.844156 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.869875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.869848 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.869991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.869956 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzgm\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-kube-api-access-drzgm\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:30.877339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.876903 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:30.877339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.876954 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:30.877652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.877636 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:30.877854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.877841 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:30.923059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.922958 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:30.945633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:30.945604 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:31.272622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.272599 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs"] Oct 30 00:14:31.274164 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:31.274134 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509d7476_b7d1_4657_b779_a2876047d78c.slice/crio-4ddff60f197056506b589fd21dfe389ce22ba2c7e262cbfae826595267334060 WatchSource:0}: Error finding container 4ddff60f197056506b589fd21dfe389ce22ba2c7e262cbfae826595267334060: Status 404 returned error can't find the container with id 4ddff60f197056506b589fd21dfe389ce22ba2c7e262cbfae826595267334060 Oct 30 00:14:31.276811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.276773 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:31.276893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.276863 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:31.276931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.276905 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:31.608966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.608937 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:31.720146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.720102 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:31.836961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.836927 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:31.883272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.883180 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" event={"ID":"509d7476-b7d1-4657-b779-a2876047d78c","Type":"ContainerStarted","Data":"77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953"} Oct 30 00:14:31.883272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.883241 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" event={"ID":"509d7476-b7d1-4657-b779-a2876047d78c","Type":"ContainerStarted","Data":"4ddff60f197056506b589fd21dfe389ce22ba2c7e262cbfae826595267334060"} Oct 30 00:14:31.934548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.934502 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podStartSLOduration=1.934487971 podStartE2EDuration="1.934487971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:31.933728051 +0000 UTC m=+580.867796605" watchObservedRunningTime="2026-10-30 00:14:31.934487971 +0000 UTC m=+580.868556506" Oct 30 00:14:31.946975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.946940 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:31.973351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:31.973325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:32.167087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:32.166983 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:32.470940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:32.470850 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:32.581893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:32.581861 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:32.951624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:32.951595 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:33.291011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:33.290916 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:33.401960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:33.401929 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:33.893933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:33.893905 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:33.894930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:33.894909 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:34.667238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.667167 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8"] Oct 30 00:14:34.669883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.669856 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.688315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.688292 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-185595da-serving-cert\"" Oct 30 00:14:34.704226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.704199 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8"] Oct 30 00:14:34.747507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.747476 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:34.781995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.781961 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznpj\" (UniqueName: \"kubernetes.io/projected/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-kube-api-access-qznpj\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.781995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.781995 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-metrics-certs\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.782161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.782076 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-openshift-service-ca\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.782212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.782200 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-maas-api-tls\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.853655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.853623 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:34.882939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.882910 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-maas-api-tls\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.883076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.882973 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznpj\" (UniqueName: \"kubernetes.io/projected/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-kube-api-access-qznpj\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.883076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.882990 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-metrics-certs\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.883076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.883023 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-openshift-service-ca\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.883762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.883740 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-openshift-service-ca\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.885398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.885373 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-maas-api-tls\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.885483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.885396 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-metrics-certs\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.908957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.908932 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznpj\" (UniqueName: \"kubernetes.io/projected/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-kube-api-access-qznpj\") pod \"maas-api-e2e-ait-185595da-6b9747cc8d-768d8\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.982216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.982146 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:34.991090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:34.991011 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:35.097868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.097837 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:35.211854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.211819 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:35.299222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.299186 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8"] Oct 30 00:14:35.301148 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:35.301118 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcafbf1_b4ed_40e6_bf53_12bf6203f883.slice/crio-574fdf7a0c3e8732f55baf0800ed12c1065474d02537c87cea2ad87e8dd97c7b WatchSource:0}: Error finding container 574fdf7a0c3e8732f55baf0800ed12c1065474d02537c87cea2ad87e8dd97c7b: Status 404 returned error can't find the container with id 574fdf7a0c3e8732f55baf0800ed12c1065474d02537c87cea2ad87e8dd97c7b Oct 30 00:14:35.833975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.833939 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:35.907206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.907168 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" event={"ID":"7dcafbf1-b4ed-40e6-bf53-12bf6203f883","Type":"ContainerStarted","Data":"6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192"} Oct 30 00:14:35.907409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.907228 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" event={"ID":"7dcafbf1-b4ed-40e6-bf53-12bf6203f883","Type":"ContainerStarted","Data":"574fdf7a0c3e8732f55baf0800ed12c1065474d02537c87cea2ad87e8dd97c7b"} Oct 30 00:14:35.907539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.907515 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:35.950399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.950351 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" podStartSLOduration=1.548466493 podStartE2EDuration="1.950337373s" totalImagesPullingTime="401.87088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:35.949957748 +0000 UTC m=+584.884026302" watchObservedRunningTime="2026-10-30 00:14:35.950337373 +0000 UTC m=+584.884405907" Oct 30 00:14:35.959975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:35.959906 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:36.201597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.201546 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:36.310858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.310818 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:36.393355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.393322 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml"] Oct 30 00:14:36.399402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.399369 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.415685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.415655 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-deda90ae-serving-cert\"" Oct 30 00:14:36.418879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.418852 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml"] Oct 30 00:14:36.482330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.482297 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:36.503603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.503544 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8k85\" (UniqueName: \"kubernetes.io/projected/066c2084-1474-4c56-a314-1787fdafefd8-kube-api-access-l8k85\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.503749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.503616 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-maas-api-tls\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.503749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.503709 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-metrics-certs\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.503835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.503814 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c2084-1474-4c56-a314-1787fdafefd8-openshift-service-ca\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.596290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.596163 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:36.605671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.605643 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c2084-1474-4c56-a314-1787fdafefd8-openshift-service-ca\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.605832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.605789 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8k85\" (UniqueName: \"kubernetes.io/projected/066c2084-1474-4c56-a314-1787fdafefd8-kube-api-access-l8k85\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.605902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.605851 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-maas-api-tls\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.605970 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.605924 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-metrics-certs\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.606425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.606399 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c2084-1474-4c56-a314-1787fdafefd8-openshift-service-ca\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.610384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.608391 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-metrics-certs\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.610384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.608544 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-maas-api-tls\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.634389 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.634252 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8k85\" (UniqueName: \"kubernetes.io/projected/066c2084-1474-4c56-a314-1787fdafefd8-kube-api-access-l8k85\") pod \"maas-api-e2e-derive-deda90ae-c59779946-qpkml\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:36.712662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:36.712628 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:37.016974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.016898 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml"] Oct 30 00:14:37.017900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.017723 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:37.020332 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:37.020305 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066c2084_1474_4c56_a314_1787fdafefd8.slice/crio-af008b3b1aad8410f9f25d80ffec77895e3c083e00965206733dbfd8ecdd5cb8 WatchSource:0}: Error finding container af008b3b1aad8410f9f25d80ffec77895e3c083e00965206733dbfd8ecdd5cb8: Status 404 returned error can't find the container with id af008b3b1aad8410f9f25d80ffec77895e3c083e00965206733dbfd8ecdd5cb8 Oct 30 00:14:37.097945 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:14:37.097905 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:14:37.147758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.147726 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:37.522967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.522939 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:37.918775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.918738 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" event={"ID":"066c2084-1474-4c56-a314-1787fdafefd8","Type":"ContainerStarted","Data":"339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293"} Oct 30 00:14:37.918969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.918791 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" event={"ID":"066c2084-1474-4c56-a314-1787fdafefd8","Type":"ContainerStarted","Data":"af008b3b1aad8410f9f25d80ffec77895e3c083e00965206733dbfd8ecdd5cb8"} Oct 30 00:14:37.918969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.918834 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:37.964530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:37.964468 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" podStartSLOduration=1.628709568 podStartE2EDuration="1.964449178s" totalImagesPullingTime="335.73961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:37.963458744 +0000 UTC m=+586.897527297" watchObservedRunningTime="2026-10-30 00:14:37.964449178 +0000 UTC m=+586.898517717" Oct 30 00:14:38.022815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.022776 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:38.163531 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.163504 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942"] Oct 30 00:14:38.166300 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.166281 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.184607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.184566 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7505bd-serving-cert\"" Oct 30 00:14:38.187384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.187348 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942"] Oct 30 00:14:38.229296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.229256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:38.322304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.322272 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f89b3b-8a1f-4b55-bff9-c22231711f78-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.322479 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.322406 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.322548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.322485 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-metrics-certs\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.322617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.322595 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w92x\" (UniqueName: \"kubernetes.io/projected/e7f89b3b-8a1f-4b55-bff9-c22231711f78-kube-api-access-8w92x\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.393880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.393847 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:38.423962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.423897 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f89b3b-8a1f-4b55-bff9-c22231711f78-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.423962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.423955 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.424170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.423999 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-metrics-certs\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.424170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.424028 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w92x\" (UniqueName: \"kubernetes.io/projected/e7f89b3b-8a1f-4b55-bff9-c22231711f78-kube-api-access-8w92x\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.424614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.424563 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f89b3b-8a1f-4b55-bff9-c22231711f78-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.426333 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.426310 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.426429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.426346 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-metrics-certs\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.447721 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.447692 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w92x\" (UniqueName: \"kubernetes.io/projected/e7f89b3b-8a1f-4b55-bff9-c22231711f78-kube-api-access-8w92x\") pod \"maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.477901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.477871 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:38.513814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.513776 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:38.785833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.785763 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942"] Oct 30 00:14:38.788084 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:38.788053 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f89b3b_8a1f_4b55_bff9_c22231711f78.slice/crio-1b7b5a2befe2895de2ea03f5f9025a1ade16dca82a8fbbfe01455167370a344d WatchSource:0}: Error finding container 1b7b5a2befe2895de2ea03f5f9025a1ade16dca82a8fbbfe01455167370a344d: Status 404 returned error can't find the container with id 1b7b5a2befe2895de2ea03f5f9025a1ade16dca82a8fbbfe01455167370a344d Oct 30 00:14:38.925040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:38.925003 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" event={"ID":"e7f89b3b-8a1f-4b55-bff9-c22231711f78","Type":"ContainerStarted","Data":"1b7b5a2befe2895de2ea03f5f9025a1ade16dca82a8fbbfe01455167370a344d"} Oct 30 00:14:39.283816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.283783 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:39.400832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.400732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:39.872961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.872931 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn"] Oct 30 00:14:39.875797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.875776 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:39.896524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.896494 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn"] Oct 30 00:14:39.903220 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.903192 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6c9e9fa7-serving-cert\"" Oct 30 00:14:39.933698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.933662 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" event={"ID":"e7f89b3b-8a1f-4b55-bff9-c22231711f78","Type":"ContainerStarted","Data":"4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f"} Oct 30 00:14:39.933873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.933798 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:39.955858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.955829 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:39.977930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:39.977880 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" podStartSLOduration=1.6779715450000001 podStartE2EDuration="1.97786537s" totalImagesPullingTime="299.893825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:39.976392523 +0000 UTC m=+588.910461057" watchObservedRunningTime="2026-10-30 00:14:39.97786537 +0000 UTC m=+588.911933905" Oct 30 00:14:40.056400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.056358 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd8779-e70c-4cd5-a4d7-7816039969df-openshift-service-ca\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.057121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.056733 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-maas-api-tls\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.057257 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.057180 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-metrics-certs\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.057343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.057294 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxrj\" (UniqueName: \"kubernetes.io/projected/7bbd8779-e70c-4cd5-a4d7-7816039969df-kube-api-access-9sxrj\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.073561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.073532 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:40.158480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.158382 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd8779-e70c-4cd5-a4d7-7816039969df-openshift-service-ca\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.158666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.158551 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-maas-api-tls\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.158731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.158689 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-metrics-certs\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.158812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.158782 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxrj\" (UniqueName: \"kubernetes.io/projected/7bbd8779-e70c-4cd5-a4d7-7816039969df-kube-api-access-9sxrj\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.159200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.159178 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd8779-e70c-4cd5-a4d7-7816039969df-openshift-service-ca\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.161004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.160980 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-maas-api-tls\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.161115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.161017 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-metrics-certs\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.250658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.250612 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxrj\" (UniqueName: \"kubernetes.io/projected/7bbd8779-e70c-4cd5-a4d7-7816039969df-kube-api-access-9sxrj\") pod \"maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.268208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.268162 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:40.488609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.488500 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:40.805728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.805702 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn"] Oct 30 00:14:40.807305 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:40.807275 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbd8779_e70c_4cd5_a4d7_7816039969df.slice/crio-7566f7af28d58f66c608d18679f1625da5989cf498765fbb52e092833e051693 WatchSource:0}: Error finding container 7566f7af28d58f66c608d18679f1625da5989cf498765fbb52e092833e051693: Status 404 returned error can't find the container with id 7566f7af28d58f66c608d18679f1625da5989cf498765fbb52e092833e051693 Oct 30 00:14:40.939750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:40.939713 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" event={"ID":"7bbd8779-e70c-4cd5-a4d7-7816039969df","Type":"ContainerStarted","Data":"7566f7af28d58f66c608d18679f1625da5989cf498765fbb52e092833e051693"} Oct 30 00:14:41.038596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.038369 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:41.174978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.169697 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:41.524298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.524214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:41.640451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.640414 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:41.878563 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.878524 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:41.918212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.918181 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:14:41.947004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.946969 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" event={"ID":"7bbd8779-e70c-4cd5-a4d7-7816039969df","Type":"ContainerStarted","Data":"8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32"} Oct 30 00:14:41.947199 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.947101 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:41.999327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:41.999294 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:42.024793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:42.024743 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" podStartSLOduration=2.681655942 podStartE2EDuration="3.024728209s" totalImagesPullingTime="343.072267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:42.024015408 +0000 UTC m=+590.958083970" watchObservedRunningTime="2026-10-30 00:14:42.024728209 +0000 UTC m=+590.958796765" Oct 30 00:14:43.785766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:43.785659 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:43.894097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:43.894063 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:43.931732 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:43.931706 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:14:45.498039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.498005 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4"] Oct 30 00:14:45.501626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.501597 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.521160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.521130 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4"] Oct 30 00:14:45.529596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.529558 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-6c45bb-serving-cert\"" Oct 30 00:14:45.546801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.546767 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71148984-2b26-4b03-86c5-8d1f56dfaf17-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.546935 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.546810 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.546992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.546951 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxxx\" (UniqueName: \"kubernetes.io/projected/71148984-2b26-4b03-86c5-8d1f56dfaf17-kube-api-access-lqxxx\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.547046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.546998 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-metrics-certs\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.595016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.594983 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:45.648532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.648499 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxxx\" (UniqueName: \"kubernetes.io/projected/71148984-2b26-4b03-86c5-8d1f56dfaf17-kube-api-access-lqxxx\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.648693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.648597 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-metrics-certs\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.648693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.648641 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71148984-2b26-4b03-86c5-8d1f56dfaf17-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.648693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.648669 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.648860 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:14:45.648831 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-serving-cert: secret "maas-api-e2e-worker-w6-6c45bb-serving-cert" not found Oct 30 00:14:45.648927 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:14:45.648915 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls podName:71148984-2b26-4b03-86c5-8d1f56dfaf17 nodeName:}" failed. No retries permitted until 2026-10-30 00:14:46.148894274 +0000 UTC m=+595.082962804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls") pod "maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" (UID: "71148984-2b26-4b03-86c5-8d1f56dfaf17") : secret "maas-api-e2e-worker-w6-6c45bb-serving-cert" not found Oct 30 00:14:45.649243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.649222 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71148984-2b26-4b03-86c5-8d1f56dfaf17-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.651091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.651072 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-metrics-certs\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.687255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.687214 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxxx\" (UniqueName: \"kubernetes.io/projected/71148984-2b26-4b03-86c5-8d1f56dfaf17-kube-api-access-lqxxx\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:45.705000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.704965 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:45.813492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.813458 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:45.946404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:45.946374 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:14:46.006148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.006115 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:46.121427 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.121342 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:46.175001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.174249 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:46.183464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.183378 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:46.418704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.418620 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:46.808909 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.808880 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4"] Oct 30 00:14:46.811265 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:14:46.811236 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71148984_2b26_4b03_86c5_8d1f56dfaf17.slice/crio-64357cc9e4dce9a291b4d897a908665248adb34027af709918b7b6889ad62a03 WatchSource:0}: Error finding container 64357cc9e4dce9a291b4d897a908665248adb34027af709918b7b6889ad62a03: Status 404 returned error can't find the container with id 64357cc9e4dce9a291b4d897a908665248adb34027af709918b7b6889ad62a03 Oct 30 00:14:46.921714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.921676 4433 patch_prober.go:29] interesting pod/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:14:46.921895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:46.921735 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:14:47.011138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:47.011094 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" event={"ID":"71148984-2b26-4b03-86c5-8d1f56dfaf17","Type":"ContainerStarted","Data":"64357cc9e4dce9a291b4d897a908665248adb34027af709918b7b6889ad62a03"} Oct 30 00:14:47.474243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:47.474059 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:48.005207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:48.005174 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:14:48.019739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:48.019692 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" event={"ID":"71148984-2b26-4b03-86c5-8d1f56dfaf17","Type":"ContainerStarted","Data":"0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227"} Oct 30 00:14:48.019928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:48.019773 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:48.105598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:48.099245 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" podStartSLOduration=2.598537951 podStartE2EDuration="3.099222604s" totalImagesPullingTime="500.684653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:48.096971491 +0000 UTC m=+597.031040027" watchObservedRunningTime="2026-10-30 00:14:48.099222604 +0000 UTC m=+597.033291139" Oct 30 00:14:49.209355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:49.209322 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:51.209100 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:51.209057 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:51.323694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:51.323657 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:14:51.324842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:51.324820 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:14:52.044446 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:52.044411 4433 generic.go:356] "Generic (PLEG): container finished" podID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerID="d36b6c07b785fecd1bd10ee8c412451e614b50ce8f7b2e227f11523e32fa5066" exitCode=0 Oct 30 00:14:52.044660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:52.044491 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" event={"ID":"f048d9a1-3b15-46b9-8309-e14d0025f24c","Type":"ContainerDied","Data":"d36b6c07b785fecd1bd10ee8c412451e614b50ce8f7b2e227f11523e32fa5066"} Oct 30 00:14:52.044949 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:52.044927 4433 scope.go:122] "RemoveContainer" containerID="d36b6c07b785fecd1bd10ee8c412451e614b50ce8f7b2e227f11523e32fa5066" Oct 30 00:14:52.046488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:52.046448 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:52.046621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:52.046539 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:52.046621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:52.046601 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:53.058623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:53.058141 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" event={"ID":"f048d9a1-3b15-46b9-8309-e14d0025f24c","Type":"ContainerStarted","Data":"7e44f7b01bcb53f2e53f61599bc9c6588ace43d49bd360d39cf243b58fc7e807"} Oct 30 00:14:53.091069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:53.091027 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:54.015657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:54.015617 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:54.015916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:54.015893 4433 patch_prober.go:29] interesting pod/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:14:54.016006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:54.015938 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 00:14:54.032141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:54.032112 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:14:54.428019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:54.427978 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:54.882893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:54.882807 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:54.999350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:54.999302 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:55.016118 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:55.016080 4433 patch_prober.go:29] interesting pod/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:14:55.016290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:55.016147 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 00:14:55.101804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:55.101769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:55.216297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:55.216212 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:56.016242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.016201 4433 patch_prober.go:29] interesting pod/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:14:56.016791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.016255 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 00:14:56.109704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.109655 4433 generic.go:356] "Generic (PLEG): container finished" podID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerID="80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c" exitCode=0 Oct 30 00:14:56.109985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.109788 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" event={"ID":"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b","Type":"ContainerDied","Data":"80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c"} Oct 30 00:14:56.110690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.110643 4433 scope.go:122] "RemoveContainer" containerID="80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c" Oct 30 00:14:56.112513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.112485 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:56.112629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.112597 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:56.112708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.112650 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:56.770655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:56.764035 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:57.015923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.015887 4433 patch_prober.go:29] interesting pod/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:14:57.016138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.015937 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 00:14:57.116120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.116093 4433 generic.go:356] "Generic (PLEG): container finished" podID="509d7476-b7d1-4657-b779-a2876047d78c" containerID="77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953" exitCode=0 Oct 30 00:14:57.116600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.116160 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" event={"ID":"509d7476-b7d1-4657-b779-a2876047d78c","Type":"ContainerDied","Data":"77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953"} Oct 30 00:14:57.117116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.117099 4433 scope.go:122] "RemoveContainer" containerID="77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953" Oct 30 00:14:57.118389 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.118367 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" event={"ID":"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b","Type":"ContainerStarted","Data":"4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95"} Oct 30 00:14:57.119383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.119354 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:57.119485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.119460 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:57.119549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.119511 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:14:57.470096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.470008 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:14:57.492081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.491718 4433 patch_prober.go:29] interesting pod/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:14:57.492081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.491783 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 00:14:57.632597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:57.632180 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:58.029745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.029708 4433 patch_prober.go:29] interesting pod/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:14:58.030083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.030048 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 00:14:58.127618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.126741 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" event={"ID":"509d7476-b7d1-4657-b779-a2876047d78c","Type":"ContainerStarted","Data":"4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4"} Oct 30 00:14:58.457600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.455939 4433 patch_prober.go:29] interesting pod/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:14:58.457600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.456018 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 00:14:58.903277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.903237 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:14:58.946619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.946588 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:14:58.946967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.946943 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:14:58.947072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:58.946996 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 00:14:59.020821 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:59.020792 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:59.021278 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:59.021253 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:59.021862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:59.021847 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:14:59.455925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:59.455889 4433 patch_prober.go:29] interesting pod/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:14:59.456508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:59.456477 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 00:14:59.947180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:59.947141 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:14:59.947369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:14:59.947196 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 00:15:00.456006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:00.455964 4433 patch_prober.go:29] interesting pod/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:15:00.456809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:00.456767 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 00:15:00.710379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:00.710210 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:00.877595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:00.876744 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:15:00.946986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:00.946954 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:15:00.947157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:00.947008 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 00:15:01.473007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:01.472970 4433 patch_prober.go:29] interesting pod/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:15:01.473611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:01.473562 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 00:15:01.947026 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:01.946992 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:15:01.947231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:01.947051 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 00:15:02.217519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:02.217434 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:02.318235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:02.318202 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:02.644547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:02.644514 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:15:02.649783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:02.649748 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:15:02.800432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:02.800395 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:02.947101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:02.947019 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:15:02.947101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:02.947069 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 00:15:03.894089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:03.894054 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:15:03.982998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:03.982946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:04.063192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.063159 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:15:04.091889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.091852 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:04.189853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.189783 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:15:04.320996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.320961 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml"] Oct 30 00:15:04.321296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.321262 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" podUID="066c2084-1474-4c56-a314-1787fdafefd8" containerName="maas-api" containerID="cri-o://339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293" gracePeriod=30 Oct 30 00:15:04.377954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.377917 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:04.625907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.625878 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:15:04.688087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.688048 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-metrics-certs\") pod \"066c2084-1474-4c56-a314-1787fdafefd8\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " Oct 30 00:15:04.688254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.688164 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066c2084-1474-4c56-a314-1787fdafefd8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c2084-1474-4c56-a314-1787fdafefd8-openshift-service-ca\") pod \"066c2084-1474-4c56-a314-1787fdafefd8\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " Oct 30 00:15:04.688254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.688194 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066c2084-1474-4c56-a314-1787fdafefd8-kube-api-access-l8k85\" (UniqueName: \"kubernetes.io/projected/066c2084-1474-4c56-a314-1787fdafefd8-kube-api-access-l8k85\") pod \"066c2084-1474-4c56-a314-1787fdafefd8\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " Oct 30 00:15:04.688254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.688228 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-maas-api-tls\") pod \"066c2084-1474-4c56-a314-1787fdafefd8\" (UID: \"066c2084-1474-4c56-a314-1787fdafefd8\") " Oct 30 00:15:04.688644 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.688604 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066c2084-1474-4c56-a314-1787fdafefd8-openshift-service-ca" pod "066c2084-1474-4c56-a314-1787fdafefd8" (UID: "066c2084-1474-4c56-a314-1787fdafefd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:04.688808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.688788 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c2084-1474-4c56-a314-1787fdafefd8-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:04.690920 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.690891 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-maas-api-tls" pod "066c2084-1474-4c56-a314-1787fdafefd8" (UID: "066c2084-1474-4c56-a314-1787fdafefd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:04.691015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.691004 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066c2084-1474-4c56-a314-1787fdafefd8-kube-api-access-l8k85" pod "066c2084-1474-4c56-a314-1787fdafefd8" (UID: "066c2084-1474-4c56-a314-1787fdafefd8"). InnerVolumeSpecName "kube-api-access-l8k85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:04.691132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.691109 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-metrics-certs" pod "066c2084-1474-4c56-a314-1787fdafefd8" (UID: "066c2084-1474-4c56-a314-1787fdafefd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:04.789873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.789782 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:04.789873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.789807 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8k85\" (UniqueName: \"kubernetes.io/projected/066c2084-1474-4c56-a314-1787fdafefd8-kube-api-access-l8k85\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:04.789873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.789816 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066c2084-1474-4c56-a314-1787fdafefd8-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:04.876241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.876201 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:04.982993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:04.982938 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:05.088789 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.088756 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:05.195054 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.195025 4433 generic.go:356] "Generic (PLEG): container finished" podID="066c2084-1474-4c56-a314-1787fdafefd8" containerID="339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293" exitCode=0 Oct 30 00:15:05.195204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.195069 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" Oct 30 00:15:05.195204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.195100 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" event={"ID":"066c2084-1474-4c56-a314-1787fdafefd8","Type":"ContainerDied","Data":"339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293"} Oct 30 00:15:05.195204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.195151 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml" event={"ID":"066c2084-1474-4c56-a314-1787fdafefd8","Type":"ContainerDied","Data":"af008b3b1aad8410f9f25d80ffec77895e3c083e00965206733dbfd8ecdd5cb8"} Oct 30 00:15:05.195204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.195173 4433 scope.go:122] "RemoveContainer" containerID="339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293" Oct 30 00:15:05.197716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.197690 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:05.204660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.204644 4433 scope.go:122] "RemoveContainer" containerID="339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293" Oct 30 00:15:05.204910 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:05.204892 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293\": container with ID starting with 339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293 not found: ID does not exist" containerID="339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293" Oct 30 00:15:05.204978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.204917 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293"} err="failed to get container status \"339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293\": rpc error: code = NotFound desc = could not find container \"339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293\": container with ID starting with 339a608b2d70aaf13c24f96e657f3a42f43d94fbc2f89fd77eb8a59c23f17293 not found: ID does not exist" Oct 30 00:15:05.253280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.253249 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml"] Oct 30 00:15:05.256729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:05.256698 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deda90ae-c59779946-qpkml"] Oct 30 00:15:06.030484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.030451 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:06.477608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.467455 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc"] Oct 30 00:15:06.477608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.469426 4433 cpu_manager.go:414] "containerMap: removing container" podUID="066c2084-1474-4c56-a314-1787fdafefd8" containerName="maas-api" Oct 30 00:15:06.477608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.469444 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066c2084-1474-4c56-a314-1787fdafefd8" containerName="maas-api" Oct 30 00:15:06.477608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.469563 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="066c2084-1474-4c56-a314-1787fdafefd8" containerName="maas-api" Oct 30 00:15:06.477608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.472668 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.488823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.488799 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b1c718-openshift-default-dockercfg-8spcf\"" Oct 30 00:15:06.505909 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.505837 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc"] Oct 30 00:15:06.523461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.523422 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:06.611962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.611925 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.611962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.611962 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.612200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.611986 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgbw\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-kube-api-access-dcgbw\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.612200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.612069 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.612200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.612180 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.612327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.612202 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acd55c2-d267-422c-9412-d6ab9621f1f0-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.612327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.612229 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.612405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.612342 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.612405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.612367 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.624784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.624755 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:06.714461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714399 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.714461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714456 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.714851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714527 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.714851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714558 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.714851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714606 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgbw\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-kube-api-access-dcgbw\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.714851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714646 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.714851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714797 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.714851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714843 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acd55c2-d267-422c-9412-d6ab9621f1f0-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.715288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.714884 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.715376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.715332 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.716304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.716274 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.716757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.716733 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.716976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.716954 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.720431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.718419 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acd55c2-d267-422c-9412-d6ab9621f1f0-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.722972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.722684 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.736738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.735196 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.736738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.736369 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:06.742048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.742011 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.743195 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.743131 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgbw\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-kube-api-access-dcgbw\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:06.788035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:06.788008 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:07.097626 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:07.097590 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:15:07.102085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.102063 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc"] Oct 30 00:15:07.103058 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:07.103036 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acd55c2_d267_422c_9412_d6ab9621f1f0.slice/crio-2e1a504f27f08fb92e3bda1821bfa9988898c580429b66848e354fb59a66830a WatchSource:0}: Error finding container 2e1a504f27f08fb92e3bda1821bfa9988898c580429b66848e354fb59a66830a: Status 404 returned error can't find the container with id 2e1a504f27f08fb92e3bda1821bfa9988898c580429b66848e354fb59a66830a Oct 30 00:15:07.105483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.105453 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:07.105538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.105518 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:07.105585 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.105548 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:07.179938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.179910 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066c2084-1474-4c56-a314-1787fdafefd8" path="/var/lib/kubelet/pods/066c2084-1474-4c56-a314-1787fdafefd8/volumes" Oct 30 00:15:07.208360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.208325 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" event={"ID":"9acd55c2-d267-422c-9412-d6ab9621f1f0","Type":"ContainerStarted","Data":"e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837"} Oct 30 00:15:07.208523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.208393 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" event={"ID":"9acd55c2-d267-422c-9412-d6ab9621f1f0","Type":"ContainerStarted","Data":"2e1a504f27f08fb92e3bda1821bfa9988898c580429b66848e354fb59a66830a"} Oct 30 00:15:07.255983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.255922 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" podStartSLOduration=1.2559072580000001 podStartE2EDuration="1.255907258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:07.25434208 +0000 UTC m=+616.188410616" watchObservedRunningTime="2026-10-30 00:15:07.255907258 +0000 UTC m=+616.189975896" Oct 30 00:15:07.789119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:07.789090 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:08.038402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:08.038367 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:08.185934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:08.185894 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:08.393259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:08.393227 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:08.497175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:08.497092 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:08.729495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:08.729457 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:08.793626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:08.793548 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:08.849347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:08.849317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:09.220367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:09.220327 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:09.221453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:09.221436 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:09.746853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:09.746808 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:09.851963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:09.851920 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:10.067101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:10.067050 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:10.818863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:10.818812 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:11.659930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:11.659893 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:11.768122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:11.768088 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:11.870922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:11.870889 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:12.490668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:12.490631 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:12.601777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:12.601745 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:12.705651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:12.705620 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:13.527910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:13.526007 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:13.964966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:13.964918 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:14.084182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:14.084154 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:14.755553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:14.755519 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8"] Oct 30 00:15:14.756010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:14.755754 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" podUID="7dcafbf1-b4ed-40e6-bf53-12bf6203f883" containerName="maas-api" containerID="cri-o://6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192" gracePeriod=30 Oct 30 00:15:14.820081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:14.819960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:15.036628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.036567 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:15:15.112325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.112295 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-maas-api-tls\") pod \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " Oct 30 00:15:15.112516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.112389 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-kube-api-access-qznpj\" (UniqueName: \"kubernetes.io/projected/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-kube-api-access-qznpj\") pod \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " Oct 30 00:15:15.112516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.112482 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-openshift-service-ca\") pod \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " Oct 30 00:15:15.112643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.112526 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-metrics-certs\") pod \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\" (UID: \"7dcafbf1-b4ed-40e6-bf53-12bf6203f883\") " Oct 30 00:15:15.113787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.113751 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-openshift-service-ca" pod "7dcafbf1-b4ed-40e6-bf53-12bf6203f883" (UID: "7dcafbf1-b4ed-40e6-bf53-12bf6203f883"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:15.115535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.115483 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-metrics-certs" pod "7dcafbf1-b4ed-40e6-bf53-12bf6203f883" (UID: "7dcafbf1-b4ed-40e6-bf53-12bf6203f883"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:15.115955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.115920 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-maas-api-tls" pod "7dcafbf1-b4ed-40e6-bf53-12bf6203f883" (UID: "7dcafbf1-b4ed-40e6-bf53-12bf6203f883"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:15.116143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.116114 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-kube-api-access-qznpj" pod "7dcafbf1-b4ed-40e6-bf53-12bf6203f883" (UID: "7dcafbf1-b4ed-40e6-bf53-12bf6203f883"). InnerVolumeSpecName "kube-api-access-qznpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:15.214505 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.214478 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:15.214505 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.214506 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznpj\" (UniqueName: \"kubernetes.io/projected/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-kube-api-access-qznpj\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:15.214719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.214520 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:15.214719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.214532 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcafbf1-b4ed-40e6-bf53-12bf6203f883-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:15.253875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.253845 4433 generic.go:356] "Generic (PLEG): container finished" podID="7dcafbf1-b4ed-40e6-bf53-12bf6203f883" containerID="6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192" exitCode=0 Oct 30 00:15:15.254031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.253891 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" Oct 30 00:15:15.254031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.253926 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" event={"ID":"7dcafbf1-b4ed-40e6-bf53-12bf6203f883","Type":"ContainerDied","Data":"6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192"} Oct 30 00:15:15.254031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.253984 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8" event={"ID":"7dcafbf1-b4ed-40e6-bf53-12bf6203f883","Type":"ContainerDied","Data":"574fdf7a0c3e8732f55baf0800ed12c1065474d02537c87cea2ad87e8dd97c7b"} Oct 30 00:15:15.254145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.254052 4433 scope.go:122] "RemoveContainer" containerID="6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192" Oct 30 00:15:15.263920 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.263901 4433 scope.go:122] "RemoveContainer" containerID="6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192" Oct 30 00:15:15.264181 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:15.264163 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192\": container with ID starting with 6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192 not found: ID does not exist" containerID="6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192" Oct 30 00:15:15.264250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.264190 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192"} err="failed to get container status \"6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192\": rpc error: code = NotFound desc = could not find container \"6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192\": container with ID starting with 6d8c11b12df700a74552567db36d1137492e28c0ad42beed20f4e48848b91192 not found: ID does not exist" Oct 30 00:15:15.312188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.312158 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:15.314069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.314043 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8"] Oct 30 00:15:15.316236 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.316213 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-185595da-6b9747cc8d-768d8"] Oct 30 00:15:15.421001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.420964 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:15.525043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:15.524967 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:16.911011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:16.910974 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn"] Oct 30 00:15:16.911450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:16.911252 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" podUID="7bbd8779-e70c-4cd5-a4d7-7816039969df" containerName="maas-api" containerID="cri-o://8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32" gracePeriod=30 Oct 30 00:15:16.963508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:16.963459 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:17.087412 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.086602 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8"] Oct 30 00:15:17.087412 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.086963 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" containerID="cri-o://7e44f7b01bcb53f2e53f61599bc9c6588ace43d49bd360d39cf243b58fc7e807" gracePeriod=30 Oct 30 00:15:17.102209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.102174 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:17.195612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.183025 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcafbf1-b4ed-40e6-bf53-12bf6203f883" path="/var/lib/kubelet/pods/7dcafbf1-b4ed-40e6-bf53-12bf6203f883/volumes" Oct 30 00:15:17.195612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.190036 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:15:17.266210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.266177 4433 generic.go:356] "Generic (PLEG): container finished" podID="7bbd8779-e70c-4cd5-a4d7-7816039969df" containerID="8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32" exitCode=0 Oct 30 00:15:17.266375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.266220 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" event={"ID":"7bbd8779-e70c-4cd5-a4d7-7816039969df","Type":"ContainerDied","Data":"8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32"} Oct 30 00:15:17.266375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.266241 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" Oct 30 00:15:17.266375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.266262 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn" event={"ID":"7bbd8779-e70c-4cd5-a4d7-7816039969df","Type":"ContainerDied","Data":"7566f7af28d58f66c608d18679f1625da5989cf498765fbb52e092833e051693"} Oct 30 00:15:17.266375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.266284 4433 scope.go:122] "RemoveContainer" containerID="8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32" Oct 30 00:15:17.277302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.277281 4433 scope.go:122] "RemoveContainer" containerID="8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32" Oct 30 00:15:17.277611 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:17.277564 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32\": container with ID starting with 8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32 not found: ID does not exist" containerID="8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32" Oct 30 00:15:17.277690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.277624 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32"} err="failed to get container status \"8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32\": rpc error: code = NotFound desc = could not find container \"8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32\": container with ID starting with 8c753e45fa21ed956528db4eba4885acb754827fddb3a5cd0acd5028265a4f32 not found: ID does not exist" Oct 30 00:15:17.334896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.334862 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-maas-api-tls\") pod \"7bbd8779-e70c-4cd5-a4d7-7816039969df\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " Oct 30 00:15:17.335065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.334943 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-metrics-certs\") pod \"7bbd8779-e70c-4cd5-a4d7-7816039969df\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " Oct 30 00:15:17.335065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.335018 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbd8779-e70c-4cd5-a4d7-7816039969df-kube-api-access-9sxrj\" (UniqueName: \"kubernetes.io/projected/7bbd8779-e70c-4cd5-a4d7-7816039969df-kube-api-access-9sxrj\") pod \"7bbd8779-e70c-4cd5-a4d7-7816039969df\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " Oct 30 00:15:17.335065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.335043 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbd8779-e70c-4cd5-a4d7-7816039969df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd8779-e70c-4cd5-a4d7-7816039969df-openshift-service-ca\") pod \"7bbd8779-e70c-4cd5-a4d7-7816039969df\" (UID: \"7bbd8779-e70c-4cd5-a4d7-7816039969df\") " Oct 30 00:15:17.335489 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.335440 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbd8779-e70c-4cd5-a4d7-7816039969df-openshift-service-ca" pod "7bbd8779-e70c-4cd5-a4d7-7816039969df" (UID: "7bbd8779-e70c-4cd5-a4d7-7816039969df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:17.337190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.337167 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-maas-api-tls" pod "7bbd8779-e70c-4cd5-a4d7-7816039969df" (UID: "7bbd8779-e70c-4cd5-a4d7-7816039969df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:17.337370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.337351 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbd8779-e70c-4cd5-a4d7-7816039969df-kube-api-access-9sxrj" pod "7bbd8779-e70c-4cd5-a4d7-7816039969df" (UID: "7bbd8779-e70c-4cd5-a4d7-7816039969df"). InnerVolumeSpecName "kube-api-access-9sxrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:17.337785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.337761 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-metrics-certs" pod "7bbd8779-e70c-4cd5-a4d7-7816039969df" (UID: "7bbd8779-e70c-4cd5-a4d7-7816039969df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:17.436906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.436827 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxrj\" (UniqueName: \"kubernetes.io/projected/7bbd8779-e70c-4cd5-a4d7-7816039969df-kube-api-access-9sxrj\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:17.436906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.436854 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd8779-e70c-4cd5-a4d7-7816039969df-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:17.436906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.436864 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:17.436906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.436872 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd8779-e70c-4cd5-a4d7-7816039969df-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:17.465965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.465932 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:17.568127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.568090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:17.614870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.614844 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn"] Oct 30 00:15:17.628382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.628352 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c9e9fa7-55f9cf4c6d-642jn"] Oct 30 00:15:17.670144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.670107 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:17.745163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.745083 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8"] Oct 30 00:15:17.745366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.745337 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" containerID="cri-o://4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95" gracePeriod=30 Oct 30 00:15:17.793306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.793277 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:17.949273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:17.949242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:18.452948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:18.452911 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:18.570689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:18.570646 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:18.686073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:18.686039 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:19.086818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.086787 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps"] Oct 30 00:15:19.087283 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.087270 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7dcafbf1-b4ed-40e6-bf53-12bf6203f883" containerName="maas-api" Oct 30 00:15:19.087327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.087284 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcafbf1-b4ed-40e6-bf53-12bf6203f883" containerName="maas-api" Oct 30 00:15:19.087327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.087297 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7bbd8779-e70c-4cd5-a4d7-7816039969df" containerName="maas-api" Oct 30 00:15:19.087327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.087302 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbd8779-e70c-4cd5-a4d7-7816039969df" containerName="maas-api" Oct 30 00:15:19.087421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.087393 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbd8779-e70c-4cd5-a4d7-7816039969df" containerName="maas-api" Oct 30 00:15:19.087421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.087409 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcafbf1-b4ed-40e6-bf53-12bf6203f883" containerName="maas-api" Oct 30 00:15:19.090416 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.090401 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.113471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.113442 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps"] Oct 30 00:15:19.127035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.124902 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-b1c718-serving-cert\"" Oct 30 00:15:19.128920 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.128893 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr"] Oct 30 00:15:19.131954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.131937 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.147528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.144512 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr"] Oct 30 00:15:19.179380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.179278 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-322d20-openshift-default-dockercfg-5q64p\"" Oct 30 00:15:19.180208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.180184 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:19.181037 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.181016 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbd8779-e70c-4cd5-a4d7-7816039969df" path="/var/lib/kubelet/pods/7bbd8779-e70c-4cd5-a4d7-7816039969df/volumes" Oct 30 00:15:19.257111 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257080 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.257262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257121 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8khg\" (UniqueName: \"kubernetes.io/projected/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-kube-api-access-m8khg\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.257262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257237 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.257360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257282 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-metrics-certs\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.257415 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257392 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34827380-65c5-479c-bf82-35dfbd7bc003-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.257470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257454 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.257528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257488 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34827380-65c5-479c-bf82-35dfbd7bc003-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.257676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257658 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.257787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257771 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.257834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.257794 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.264015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.258172 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.264015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.258286 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.264015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.258341 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdp9v\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-kube-api-access-fdp9v\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.294315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.294290 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:19.359530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359453 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.359530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359493 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.359530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359517 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.359828 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359633 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.359828 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359678 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdp9v\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-kube-api-access-fdp9v\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.359828 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359737 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.359828 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359774 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8khg\" (UniqueName: \"kubernetes.io/projected/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-kube-api-access-m8khg\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.360040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359855 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359892 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-metrics-certs\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.360040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359934 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34827380-65c5-479c-bf82-35dfbd7bc003-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359942 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359964 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.359981 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.360040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.360017 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34827380-65c5-479c-bf82-35dfbd7bc003-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.360117 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.360176 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360620 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.360595 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.360832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.360812 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.361261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.361239 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34827380-65c5-479c-bf82-35dfbd7bc003-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.362545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.362504 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.362698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.362675 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34827380-65c5-479c-bf82-35dfbd7bc003-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.362763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.362675 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.362886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.362868 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-metrics-certs\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.386546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.386519 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.387444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.387421 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8khg\" (UniqueName: \"kubernetes.io/projected/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-kube-api-access-m8khg\") pod \"maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.387708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.387692 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdp9v\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-kube-api-access-fdp9v\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.402277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.402257 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:19.403307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.403283 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:19.462687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.456892 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:19.700907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.700880 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps"] Oct 30 00:15:19.702471 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:19.702446 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1ef643_a2f0_4cd2_8788_08b8084b0dca.slice/crio-3e6fdd8b0c4699b4a17bd65387d73b9bde16a0bbb492cdc2ebdc963280e108ee WatchSource:0}: Error finding container 3e6fdd8b0c4699b4a17bd65387d73b9bde16a0bbb492cdc2ebdc963280e108ee: Status 404 returned error can't find the container with id 3e6fdd8b0c4699b4a17bd65387d73b9bde16a0bbb492cdc2ebdc963280e108ee Oct 30 00:15:19.703969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.703951 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:15:19.807850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.807820 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:19.815769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.815744 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr"] Oct 30 00:15:19.817421 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:19.817389 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34827380_65c5_479c_bf82_35dfbd7bc003.slice/crio-6646511561df5e34514e4499ff5581d7cb00a090bfd1853884f29a2777a5259d WatchSource:0}: Error finding container 6646511561df5e34514e4499ff5581d7cb00a090bfd1853884f29a2777a5259d: Status 404 returned error can't find the container with id 6646511561df5e34514e4499ff5581d7cb00a090bfd1853884f29a2777a5259d Oct 30 00:15:19.820097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.820058 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:19.820187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.820135 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:19.820187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:19.820172 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:20.286436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.286397 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" event={"ID":"6f1ef643-a2f0-4cd2-8788-08b8084b0dca","Type":"ContainerStarted","Data":"2ad4b50f0bdaabe141add6aa6357f1a23d0caab451c4fcfb46f7bb83bd562043"} Oct 30 00:15:20.286954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.286458 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" event={"ID":"6f1ef643-a2f0-4cd2-8788-08b8084b0dca","Type":"ContainerStarted","Data":"3e6fdd8b0c4699b4a17bd65387d73b9bde16a0bbb492cdc2ebdc963280e108ee"} Oct 30 00:15:20.286954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.286606 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:20.288101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.288073 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" event={"ID":"34827380-65c5-479c-bf82-35dfbd7bc003","Type":"ContainerStarted","Data":"6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee"} Oct 30 00:15:20.288206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.288120 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" event={"ID":"34827380-65c5-479c-bf82-35dfbd7bc003","Type":"ContainerStarted","Data":"6646511561df5e34514e4499ff5581d7cb00a090bfd1853884f29a2777a5259d"} Oct 30 00:15:20.376788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.376736 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" podStartSLOduration=1.040276307 podStartE2EDuration="1.376719012s" totalImagesPullingTime="336.442705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:20.373260157 +0000 UTC m=+629.307328708" watchObservedRunningTime="2026-10-30 00:15:20.376719012 +0000 UTC m=+629.310787547" Oct 30 00:15:20.441835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.441781 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" podStartSLOduration=1.441766097 podStartE2EDuration="1.441766097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:20.441644641 +0000 UTC m=+629.375713169" watchObservedRunningTime="2026-10-30 00:15:20.441766097 +0000 UTC m=+629.375834632" Oct 30 00:15:20.457840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.457815 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:20.462824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.462800 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:20.994432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:20.994402 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:21.294016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:21.293931 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:21.294701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:21.294681 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:15:22.301833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.301800 4433 generic.go:356] "Generic (PLEG): container finished" podID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerID="7e44f7b01bcb53f2e53f61599bc9c6588ace43d49bd360d39cf243b58fc7e807" exitCode=0 Oct 30 00:15:22.302248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.301865 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" event={"ID":"f048d9a1-3b15-46b9-8309-e14d0025f24c","Type":"ContainerDied","Data":"7e44f7b01bcb53f2e53f61599bc9c6588ace43d49bd360d39cf243b58fc7e807"} Oct 30 00:15:22.302248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.301928 4433 scope.go:122] "RemoveContainer" containerID="d36b6c07b785fecd1bd10ee8c412451e614b50ce8f7b2e227f11523e32fa5066" Oct 30 00:15:22.354378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.354322 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:15:22.443023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.442987 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:22.501477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501446 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-socket\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501497 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-data\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501530 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-envoy\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501561 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f048d9a1-3b15-46b9-8309-e14d0025f24c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f048d9a1-3b15-46b9-8309-e14d0025f24c-istiod-ca-cert\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501620 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-token\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-token\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501653 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-kube-api-access-zp9gq\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-kube-api-access-zp9gq\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501687 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-podinfo\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501797 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-certs\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.501931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501779 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-socket" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:22.501931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501841 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-data" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:22.501931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.501880 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-credential-socket\") pod \"f048d9a1-3b15-46b9-8309-e14d0025f24c\" (UID: \"f048d9a1-3b15-46b9-8309-e14d0025f24c\") " Oct 30 00:15:22.502180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502101 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f048d9a1-3b15-46b9-8309-e14d0025f24c-istiod-ca-cert" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:22.502294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502186 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-certs" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:22.502360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502299 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-credential-socket" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:22.502465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502449 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.502528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502465 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.502528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502474 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f048d9a1-3b15-46b9-8309-e14d0025f24c-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.502528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502483 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.502528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.502492 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.503916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.503893 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-envoy" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:22.504315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.504298 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-token" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:22.504392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.504320 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-kube-api-access-zp9gq" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "kube-api-access-zp9gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:22.504480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.504460 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-podinfo" pod "f048d9a1-3b15-46b9-8309-e14d0025f24c" (UID: "f048d9a1-3b15-46b9-8309-e14d0025f24c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:15:22.576379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.576348 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:22.604102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.604077 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.604102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.604101 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp9gq\" (UniqueName: \"kubernetes.io/projected/f048d9a1-3b15-46b9-8309-e14d0025f24c-kube-api-access-zp9gq\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.604276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.604110 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.604276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.604120 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f048d9a1-3b15-46b9-8309-e14d0025f24c-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:22.697747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.697656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:22.945591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.945547 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:22.987690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:22.987668 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:15:23.009222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009195 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istiod-ca-cert\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009233 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-certs\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009261 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-credential-socket\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009356 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-podinfo\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009384 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-envoy\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009407 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-token\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-token\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009440 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-socket\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009481 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-data\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.009800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009596 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-certs" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:23.009800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009679 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istiod-ca-cert" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:23.009921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009823 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-socket" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:23.009921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009837 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-data" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:23.009921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.009876 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-credential-socket" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:23.010253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.010231 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.010341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.010258 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.010341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.010272 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.010341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.010283 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.010341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.010296 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.011707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.011682 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-podinfo" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:15:23.011794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.011705 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-token" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:23.011794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.011741 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-envoy" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:15:23.107654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.107621 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:23.110802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.110774 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-kube-api-access-h9ltl\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-kube-api-access-h9ltl\") pod \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\" (UID: \"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b\") " Oct 30 00:15:23.111233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.111211 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.111331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.111236 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.111331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.111250 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.112895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.112869 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-kube-api-access-h9ltl" pod "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" (UID: "be1bcfc7-851d-48c7-bc90-3e8e69a36e2b"). InnerVolumeSpecName "kube-api-access-h9ltl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:23.210367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.210330 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:23.212777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.212751 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ltl\" (UniqueName: \"kubernetes.io/projected/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b-kube-api-access-h9ltl\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:15:23.309679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.309646 4433 generic.go:356] "Generic (PLEG): container finished" podID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerID="4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95" exitCode=0 Oct 30 00:15:23.310091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.309721 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" event={"ID":"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b","Type":"ContainerDied","Data":"4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95"} Oct 30 00:15:23.310091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.309761 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" Oct 30 00:15:23.310091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.309800 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8" event={"ID":"be1bcfc7-851d-48c7-bc90-3e8e69a36e2b","Type":"ContainerDied","Data":"1f988daa5db3c2c5cd1f44fb1142dadab9c29db02fad57e752e0e71462ba2fcc"} Oct 30 00:15:23.310091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.309836 4433 scope.go:122] "RemoveContainer" containerID="4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95" Oct 30 00:15:23.312003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.311967 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" Oct 30 00:15:23.312484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.312035 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8" event={"ID":"f048d9a1-3b15-46b9-8309-e14d0025f24c","Type":"ContainerDied","Data":"1a81b17e4b276a12e73a2ea59cbb398c2fff0d286795d77b59203abe556a395b"} Oct 30 00:15:23.320897 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.320877 4433 scope.go:122] "RemoveContainer" containerID="80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c" Oct 30 00:15:23.330297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.330280 4433 scope.go:122] "RemoveContainer" containerID="4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95" Oct 30 00:15:23.330603 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:23.330562 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95\": container with ID starting with 4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95 not found: ID does not exist" containerID="4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95" Oct 30 00:15:23.330742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.330615 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95"} err="failed to get container status \"4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95\": rpc error: code = NotFound desc = could not find container \"4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95\": container with ID starting with 4b185fc6150292d12f0c1d1297f1789e6a5452bde4cef19641764608a814ac95 not found: ID does not exist" Oct 30 00:15:23.330742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.330640 4433 scope.go:122] "RemoveContainer" containerID="80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c" Oct 30 00:15:23.330742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.330676 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:23.330951 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:23.330928 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c\": container with ID starting with 80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c not found: ID does not exist" containerID="80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c" Oct 30 00:15:23.330996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.330957 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c"} err="failed to get container status \"80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c\": rpc error: code = NotFound desc = could not find container \"80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c\": container with ID starting with 80a02da8f4ca7627e92cc9a559801a930ac69afb92dcaeb5c47ee29517febe7c not found: ID does not exist" Oct 30 00:15:23.330996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.330978 4433 scope.go:122] "RemoveContainer" containerID="7e44f7b01bcb53f2e53f61599bc9c6588ace43d49bd360d39cf243b58fc7e807" Oct 30 00:15:23.367366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.367341 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8"] Oct 30 00:15:23.379452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.379431 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-185595da-openshift-default-58fd56f8fc-xjxf8"] Oct 30 00:15:23.418044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.418016 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8"] Oct 30 00:15:23.422866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.422842 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6c9e9fa7-openshift-default-7568575669-hc6r8"] Oct 30 00:15:23.469936 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.469901 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:23.690508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.690418 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr"] Oct 30 00:15:23.690954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.690933 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" Oct 30 00:15:23.690954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.690950 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" Oct 30 00:15:23.690954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.690959 4433 cpu_manager.go:414] "containerMap: removing container" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" Oct 30 00:15:23.691163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.690964 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" Oct 30 00:15:23.691163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.690975 4433 cpu_manager.go:414] "containerMap: removing container" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" Oct 30 00:15:23.691163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.690980 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" Oct 30 00:15:23.691163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.691078 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" Oct 30 00:15:23.691163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.691087 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" Oct 30 00:15:23.694432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.694410 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.713078 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.713056 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"kube-root-ca.crt\"" Oct 30 00:15:23.713181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.713097 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"openshift-service-ca.crt\"" Oct 30 00:15:23.715458 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.715435 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr"] Oct 30 00:15:23.717008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.716990 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"simulator-w1-7a7ac6-kserve-self-signed-certs\"" Oct 30 00:15:23.717107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.717031 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"default-dockercfg-hvhpt\"" Oct 30 00:15:23.717174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.717143 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-model-cache\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.717232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.717171 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-tmp-dir\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.717232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.717189 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-home\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.717332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.717306 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-dshm\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.717388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.717377 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93a188e-e62f-4e56-b197-87d5361ab3ea-tls-certs\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.717440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.717408 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbc5t\" (UniqueName: \"kubernetes.io/projected/b93a188e-e62f-4e56-b197-87d5361ab3ea-kube-api-access-mbc5t\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.744237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.744207 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:23.818651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.818612 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-dshm\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.818816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.818770 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93a188e-e62f-4e56-b197-87d5361ab3ea-tls-certs\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.818816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.818799 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbc5t\" (UniqueName: \"kubernetes.io/projected/b93a188e-e62f-4e56-b197-87d5361ab3ea-kube-api-access-mbc5t\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.818949 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.818898 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-model-cache\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.818949 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.818924 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-tmp-dir\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.819088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.819062 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-home\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.819313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.819281 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-model-cache\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.819418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.819357 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-tmp-dir\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.819418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.819382 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-home\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.821184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.821161 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-dshm\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.821457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.821441 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93a188e-e62f-4e56-b197-87d5361ab3ea-tls-certs\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:23.842840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:23.842809 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbc5t\" (UniqueName: \"kubernetes.io/projected/b93a188e-e62f-4e56-b197-87d5361ab3ea-kube-api-access-mbc5t\") pod \"simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:24.005634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:24.005545 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:24.191514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:24.191481 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:24.300357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:24.300264 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:24.312872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:24.312845 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr"] Oct 30 00:15:24.314645 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:24.314618 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93a188e_e62f_4e56_b197_87d5361ab3ea.slice/crio-8de3c4dd5b350a7efb56e87f4484e5c256c847e6b45c0099c28fa1dd3991fd9a WatchSource:0}: Error finding container 8de3c4dd5b350a7efb56e87f4484e5c256c847e6b45c0099c28fa1dd3991fd9a: Status 404 returned error can't find the container with id 8de3c4dd5b350a7efb56e87f4484e5c256c847e6b45c0099c28fa1dd3991fd9a Oct 30 00:15:24.407062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:24.407029 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:24.787298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:24.786549 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:24.891667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:24.891633 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:25.179101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:25.179067 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" path="/var/lib/kubelet/pods/be1bcfc7-851d-48c7-bc90-3e8e69a36e2b/volumes" Oct 30 00:15:25.179554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:25.179539 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" path="/var/lib/kubelet/pods/f048d9a1-3b15-46b9-8309-e14d0025f24c/volumes" Oct 30 00:15:25.193488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:25.193460 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:25.327700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:25.327663 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" event={"ID":"b93a188e-e62f-4e56-b197-87d5361ab3ea","Type":"ContainerStarted","Data":"5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0"} Oct 30 00:15:25.328184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:25.327728 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" event={"ID":"b93a188e-e62f-4e56-b197-87d5361ab3ea","Type":"ContainerStarted","Data":"8de3c4dd5b350a7efb56e87f4484e5c256c847e6b45c0099c28fa1dd3991fd9a"} Oct 30 00:15:25.371240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:25.371186 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podStartSLOduration=2.371173617 podStartE2EDuration="2.371173617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:25.370302272 +0000 UTC m=+634.304370809" watchObservedRunningTime="2026-10-30 00:15:25.371173617 +0000 UTC m=+634.305242361" Oct 30 00:15:25.975936 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:25.975905 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:26.005931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:26.005899 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:26.077421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:26.077390 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:26.310439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:26.310410 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:15:26.901282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:26.901241 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:28.687489 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:28.687454 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:28.901893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:28.901855 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:30.415704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:30.415669 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:30.527833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:30.527794 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:30.643010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:30.642974 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:31.796835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.796798 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd"] Oct 30 00:15:31.806710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.806680 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" Oct 30 00:15:31.806934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.806917 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" Oct 30 00:15:31.807268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.807255 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1bcfc7-851d-48c7-bc90-3e8e69a36e2b" containerName="istio-proxy" Oct 30 00:15:31.807357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.807347 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f048d9a1-3b15-46b9-8309-e14d0025f24c" containerName="istio-proxy" Oct 30 00:15:31.811303 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.811280 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.814179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.814153 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-model-cache\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.814414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.814395 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-tmp-dir\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.814884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.814866 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af12066d-62f8-49e1-a886-7eb0b1bfae07-tls-certs\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.815271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.815146 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-home\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.815455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.815436 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k94xh\" (UniqueName: \"kubernetes.io/projected/af12066d-62f8-49e1-a886-7eb0b1bfae07-kube-api-access-k94xh\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.815687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.815669 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-dshm\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.831598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.831501 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd"] Oct 30 00:15:31.838493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.838468 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"kube-root-ca.crt\"" Oct 30 00:15:31.838493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.838470 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"openshift-service-ca.crt\"" Oct 30 00:15:31.838675 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.838498 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"default-dockercfg-z7289\"" Oct 30 00:15:31.844480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.844464 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"premium-w6-6c45bb-kserve-self-signed-certs\"" Oct 30 00:15:31.883067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.882887 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:31.916891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.916858 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-home\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.916899 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k94xh\" (UniqueName: \"kubernetes.io/projected/af12066d-62f8-49e1-a886-7eb0b1bfae07-kube-api-access-k94xh\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.916953 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-dshm\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.916985 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-model-cache\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.917000 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-tmp-dir\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.917072 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af12066d-62f8-49e1-a886-7eb0b1bfae07-tls-certs\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917394 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.917362 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-home\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.917425 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-tmp-dir\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.917521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.917463 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-model-cache\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.919195 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.919176 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-dshm\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.919484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.919464 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af12066d-62f8-49e1-a886-7eb0b1bfae07-tls-certs\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:31.940976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:31.940952 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k94xh\" (UniqueName: \"kubernetes.io/projected/af12066d-62f8-49e1-a886-7eb0b1bfae07-kube-api-access-k94xh\") pod \"premium-w6-6c45bb-kserve-8d4fc8b48-frlmd\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:32.143779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:32.143736 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:32.483040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:32.482960 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd"] Oct 30 00:15:32.485549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:32.485514 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:32.488862 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:32.488833 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf12066d_62f8_49e1_a886_7eb0b1bfae07.slice/crio-939533f767e8230859893a831efb0c6fd5518e865d24ffde78ec7785104397ca WatchSource:0}: Error finding container 939533f767e8230859893a831efb0c6fd5518e865d24ffde78ec7785104397ca: Status 404 returned error can't find the container with id 939533f767e8230859893a831efb0c6fd5518e865d24ffde78ec7785104397ca Oct 30 00:15:32.585353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:32.585317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:32.688124 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:32.687956 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:33.409855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:33.409814 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" event={"ID":"af12066d-62f8-49e1-a886-7eb0b1bfae07","Type":"ContainerStarted","Data":"e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e"} Oct 30 00:15:33.409855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:33.409859 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" event={"ID":"af12066d-62f8-49e1-a886-7eb0b1bfae07","Type":"ContainerStarted","Data":"939533f767e8230859893a831efb0c6fd5518e865d24ffde78ec7785104397ca"} Oct 30 00:15:33.453698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:33.453636 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podStartSLOduration=2.453618387 podStartE2EDuration="2.453618387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:33.451749583 +0000 UTC m=+642.385818119" watchObservedRunningTime="2026-10-30 00:15:33.453618387 +0000 UTC m=+642.387686923" Oct 30 00:15:34.006166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.006128 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:34.020441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.020409 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:34.106785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.106749 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp"] Oct 30 00:15:34.110046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.110023 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.132839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.132795 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp"] Oct 30 00:15:34.133963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.133940 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-322d20-serving-cert\"" Oct 30 00:15:34.142967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.142934 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-metrics-certs\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.143118 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.143007 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-maas-api-tls\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.143118 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.143088 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtjl\" (UniqueName: \"kubernetes.io/projected/c9ad66e9-958f-4411-9c86-542cb7739541-kube-api-access-cgtjl\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.143239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.143161 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad66e9-958f-4411-9c86-542cb7739541-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.144506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.144481 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:34.217902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.217865 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:34.244267 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.244234 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-metrics-certs\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.244456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.244288 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-maas-api-tls\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.244456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.244309 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtjl\" (UniqueName: \"kubernetes.io/projected/c9ad66e9-958f-4411-9c86-542cb7739541-kube-api-access-cgtjl\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.244456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.244356 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad66e9-958f-4411-9c86-542cb7739541-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.245210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.245180 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad66e9-958f-4411-9c86-542cb7739541-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.246998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.246975 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-maas-api-tls\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.247705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.247676 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-metrics-certs\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.268418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.268300 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtjl\" (UniqueName: \"kubernetes.io/projected/c9ad66e9-958f-4411-9c86-542cb7739541-kube-api-access-cgtjl\") pod \"maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.323122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.323082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:34.423857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.423826 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:34.431706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.431678 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:15:34.726547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.726517 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp"] Oct 30 00:15:34.728775 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:34.728749 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ad66e9_958f_4411_9c86_542cb7739541.slice/crio-904e78be195e00a2cc9bae890d6310312da661ad4557e93e596789b08364a74b WatchSource:0}: Error finding container 904e78be195e00a2cc9bae890d6310312da661ad4557e93e596789b08364a74b: Status 404 returned error can't find the container with id 904e78be195e00a2cc9bae890d6310312da661ad4557e93e596789b08364a74b Oct 30 00:15:34.946305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:34.946267 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:35.426012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.425916 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" event={"ID":"c9ad66e9-958f-4411-9c86-542cb7739541","Type":"ContainerStarted","Data":"ebe1a154dff80bf3f7342a7b0bab9dc5dc0364eb60b54b8c6209b3cc90bfb168"} Oct 30 00:15:35.426012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.425981 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" event={"ID":"c9ad66e9-958f-4411-9c86-542cb7739541","Type":"ContainerStarted","Data":"904e78be195e00a2cc9bae890d6310312da661ad4557e93e596789b08364a74b"} Oct 30 00:15:35.426497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.426261 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:35.477905 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.477021 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" podStartSLOduration=1.10251932 podStartE2EDuration="1.477001812s" totalImagesPullingTime="374.482492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:35.46963514 +0000 UTC m=+644.403703674" watchObservedRunningTime="2026-10-30 00:15:35.477001812 +0000 UTC m=+644.411070349" Oct 30 00:15:35.582528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.582488 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:35.699939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.699849 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:35.815624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.814652 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:35.953849 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:35.953744 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:36.327264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:36.327231 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:36.433945 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:36.433866 4433 generic.go:356] "Generic (PLEG): container finished" podID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerID="e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837" exitCode=0 Oct 30 00:15:36.434314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:36.433939 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" event={"ID":"9acd55c2-d267-422c-9412-d6ab9621f1f0","Type":"ContainerDied","Data":"e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837"} Oct 30 00:15:36.434449 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:36.434434 4433 scope.go:122] "RemoveContainer" containerID="e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837" Oct 30 00:15:36.435812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:36.435785 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:36.435866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:36.435849 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:36.435900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:36.435880 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:37.102780 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:37.102746 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:15:37.441809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:37.441730 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" event={"ID":"9acd55c2-d267-422c-9412-d6ab9621f1f0","Type":"ContainerStarted","Data":"e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044"} Oct 30 00:15:37.512027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:37.511984 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:37.788838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:37.788762 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:37.789944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:37.789921 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:15:37.790066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:37.789968 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 00:15:38.789008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:38.788971 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:15:38.789396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:38.789022 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 00:15:39.207921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:39.207892 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:39.220782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:39.220756 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:39.788660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:39.788622 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:15:39.788832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:39.788680 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 00:15:40.773782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.773711 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk"] Oct 30 00:15:40.798778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.798743 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:15:40.799073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.799045 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 00:15:40.800782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.800727 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.804359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.804327 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk"] Oct 30 00:15:40.819200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.819173 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-564c8f-serving-cert\"" Oct 30 00:15:40.820904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.820878 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.821127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.821104 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.821176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.821143 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-metrics-certs\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.821324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.821299 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6r6\" (UniqueName: \"kubernetes.io/projected/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-kube-api-access-lh6r6\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.846206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.846171 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:40.922221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.922188 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.922221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.922225 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-metrics-certs\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.922436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.922282 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6r6\" (UniqueName: \"kubernetes.io/projected/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-kube-api-access-lh6r6\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.922436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.922318 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.922436 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:40.922416 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-serving-cert: secret "maas-api-e2e-worker-w3-564c8f-serving-cert" not found Oct 30 00:15:40.922566 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:15:40.922480 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls podName:a98846a5-27d4-4a29-82ce-a573f0a1a3eb nodeName:}" failed. No retries permitted until 2026-10-30 00:15:41.422465228 +0000 UTC m=+650.356533741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls") pod "maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" (UID: "a98846a5-27d4-4a29-82ce-a573f0a1a3eb") : secret "maas-api-e2e-worker-w3-564c8f-serving-cert" not found Oct 30 00:15:40.922985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.922963 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.924971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.924944 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-metrics-certs\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:40.949041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.948999 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:40.950513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:40.950482 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6r6\" (UniqueName: \"kubernetes.io/projected/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-kube-api-access-lh6r6\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:41.140060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:41.140021 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:41.427472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:41.427370 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:41.430171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:41.430144 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:41.439990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:41.439970 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:15:41.717549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:41.717462 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:41.788835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:41.788806 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:15:41.789250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:41.788874 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 00:15:42.036869 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.036841 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk"] Oct 30 00:15:42.038932 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:42.038900 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98846a5_27d4_4a29_82ce_a573f0a1a3eb.slice/crio-2463d3a13cd42c2f3d803dbc42cbeba6b54032a4f41d148e520b95d494dd9010 WatchSource:0}: Error finding container 2463d3a13cd42c2f3d803dbc42cbeba6b54032a4f41d148e520b95d494dd9010: Status 404 returned error can't find the container with id 2463d3a13cd42c2f3d803dbc42cbeba6b54032a4f41d148e520b95d494dd9010 Oct 30 00:15:42.144518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.144472 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:42.184197 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.184165 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:42.491340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.490689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:42.493731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.493698 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" event={"ID":"a98846a5-27d4-4a29-82ce-a573f0a1a3eb","Type":"ContainerStarted","Data":"2463d3a13cd42c2f3d803dbc42cbeba6b54032a4f41d148e520b95d494dd9010"} Oct 30 00:15:42.510460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.510422 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:15:42.794238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.794156 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:42.795352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:42.795331 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:15:43.505528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:43.505481 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" event={"ID":"a98846a5-27d4-4a29-82ce-a573f0a1a3eb","Type":"ContainerStarted","Data":"297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e"} Oct 30 00:15:43.505762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:43.505745 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:43.549884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:43.549834 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" podStartSLOduration=3.158061851 podStartE2EDuration="3.549821536s" totalImagesPullingTime="391.759685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:43.548544645 +0000 UTC m=+652.482613180" watchObservedRunningTime="2026-10-30 00:15:43.549821536 +0000 UTC m=+652.483890072" Oct 30 00:15:44.083061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:44.083017 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:45.560343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:45.560304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:46.636320 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.636286 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w"] Oct 30 00:15:46.639364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.639341 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.670953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.670846 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"openshift-service-ca.crt\"" Oct 30 00:15:46.670953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.670885 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"default-dockercfg-q56cf\"" Oct 30 00:15:46.670953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.670958 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"simulator-w5-b1c718-kserve-self-signed-certs\"" Oct 30 00:15:46.671209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.671043 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"kube-root-ca.crt\"" Oct 30 00:15:46.675053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.675031 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w"] Oct 30 00:15:46.686379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.686350 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:46.698355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.698323 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5v66\" (UniqueName: \"kubernetes.io/projected/dc58932a-ae73-4b75-862b-30ebbdebfeb4-kube-api-access-l5v66\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.698516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.698384 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-dshm\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.698516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.698449 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-home\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.698630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.698528 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tmp-dir\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.698630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.698606 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-model-cache\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.698736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.698704 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tls-certs\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.800325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800290 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-dshm\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.800325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800328 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-home\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.801108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800369 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tmp-dir\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.801108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800431 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-model-cache\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.801108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800459 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tls-certs\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.801108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800531 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5v66\" (UniqueName: \"kubernetes.io/projected/dc58932a-ae73-4b75-862b-30ebbdebfeb4-kube-api-access-l5v66\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.801108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800916 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-home\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.801108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.800953 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tmp-dir\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.801108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.801002 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-model-cache\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.802674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.802654 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-dshm\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.802966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.802949 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tls-certs\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.824469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.824434 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5v66\" (UniqueName: \"kubernetes.io/projected/dc58932a-ae73-4b75-862b-30ebbdebfeb4-kube-api-access-l5v66\") pod \"simulator-w5-b1c718-kserve-6855bb954-k4w9w\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:46.951241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:46.951158 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:47.064968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:47.064929 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:47.253884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:47.253856 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w"] Oct 30 00:15:47.255685 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:15:47.255653 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc58932a_ae73_4b75_862b_30ebbdebfeb4.slice/crio-a7bbd524ac076446adaa5803c7046d2db66c0c24571847ebb4f8a150de93554d WatchSource:0}: Error finding container a7bbd524ac076446adaa5803c7046d2db66c0c24571847ebb4f8a150de93554d: Status 404 returned error can't find the container with id a7bbd524ac076446adaa5803c7046d2db66c0c24571847ebb4f8a150de93554d Oct 30 00:15:47.529996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:47.529900 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" event={"ID":"dc58932a-ae73-4b75-862b-30ebbdebfeb4","Type":"ContainerStarted","Data":"c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c"} Oct 30 00:15:47.529996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:47.529958 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" event={"ID":"dc58932a-ae73-4b75-862b-30ebbdebfeb4","Type":"ContainerStarted","Data":"a7bbd524ac076446adaa5803c7046d2db66c0c24571847ebb4f8a150de93554d"} Oct 30 00:15:47.575848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:47.575784 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podStartSLOduration=1.57576858 podStartE2EDuration="1.57576858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:47.573714514 +0000 UTC m=+656.507783050" watchObservedRunningTime="2026-10-30 00:15:47.57576858 +0000 UTC m=+656.509837135" Oct 30 00:15:47.818628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:47.818592 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:47.951341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:47.951303 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:48.502120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:48.502081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:48.619586 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:48.619535 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:48.725624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:48.725561 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:48.925663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:48.925541 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:49.518388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:49.518357 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:15:50.606259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:50.606227 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:52.324815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:52.324776 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:52.832999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:52.832958 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:53.328294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:53.328255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:53.978749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:53.978710 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:54.086002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:54.085957 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:54.198252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:54.198157 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:55.677413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:55.677372 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:56.952061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:56.952021 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:56.964825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:56.964794 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:57.377625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:57.377592 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:57.630953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:57.630867 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:15:58.627618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:58.627553 4433 generic.go:356] "Generic (PLEG): container finished" podID="34827380-65c5-479c-bf82-35dfbd7bc003" containerID="6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee" exitCode=0 Oct 30 00:15:58.627618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:58.627610 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" event={"ID":"34827380-65c5-479c-bf82-35dfbd7bc003","Type":"ContainerDied","Data":"6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee"} Oct 30 00:15:58.628228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:58.628214 4433 scope.go:122] "RemoveContainer" containerID="6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee" Oct 30 00:15:58.629626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:58.629563 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:58.629714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:58.629646 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:58.629714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:58.629679 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:15:59.371449 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:59.371414 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:59.636763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:59.636684 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" event={"ID":"34827380-65c5-479c-bf82-35dfbd7bc003","Type":"ContainerStarted","Data":"192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6"} Oct 30 00:15:59.676598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:59.673603 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:59.782518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:59.782488 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:15:59.889701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:15:59.889569 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:00.473961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:00.473929 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:16:00.474294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:00.474272 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:16:00.474446 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:00.474422 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:16:00.835396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:00.835363 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:01.278812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:01.278709 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:01.463758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:01.463723 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:16:01.478613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:01.478559 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:16:02.463951 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:02.463914 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:16:02.465473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:02.465441 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:16:02.612762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:02.612722 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:03.474002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:03.473961 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:16:03.475703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:03.475661 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:16:04.306406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:04.306362 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:04.471597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:04.464959 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:16:04.471597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:04.465022 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:16:05.462156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:05.462078 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:16:05.462597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:05.462499 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:16:05.463180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:05.463159 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:16:05.870867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:05.870824 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:07.098076 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:16:07.098038 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:16:07.379366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:07.379283 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:08.692896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.692867 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx"] Oct 30 00:16:08.695681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.695660 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.712117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.712091 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx"] Oct 30 00:16:08.721488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.721456 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"openshift-service-ca.crt\"" Oct 30 00:16:08.721720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.721469 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"kube-root-ca.crt\"" Oct 30 00:16:08.724223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.724205 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"default-dockercfg-kk5f8\"" Oct 30 00:16:08.724313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.724254 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"simulator-w2-322d20-kserve-self-signed-certs\"" Oct 30 00:16:08.739802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.739775 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:08.784945 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.784911 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-model-cache\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.784945 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.784944 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlgx\" (UniqueName: \"kubernetes.io/projected/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-kube-api-access-htlgx\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.785178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.784972 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-home\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.785178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.785047 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tmp-dir\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.785266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.785250 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tls-certs\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.785318 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.785305 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-dshm\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.886645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.886609 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-model-cache\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.886833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.886653 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlgx\" (UniqueName: \"kubernetes.io/projected/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-kube-api-access-htlgx\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.886833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.886681 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-home\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.886833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.886700 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tmp-dir\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.886998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.886864 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tls-certs\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.886998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.886946 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-dshm\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.886998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.886980 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-model-cache\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.887125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.887062 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-home\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.887174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.887151 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tmp-dir\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.889096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.889073 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-dshm\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.889243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.889224 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tls-certs\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.910746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.910720 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlgx\" (UniqueName: \"kubernetes.io/projected/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-kube-api-access-htlgx\") pod \"simulator-w2-322d20-kserve-799c4dccf4-52whx\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:08.998932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:08.998851 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:09.009745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:09.009718 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:09.314173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:09.314148 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx"] Oct 30 00:16:09.316070 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:16:09.316046 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4deb7d9_4a19_418f_a8c3_9d3064257bfd.slice/crio-001ce20f153953e3e5340f690fe5fefc372802be3894966e765bc434e97e4f1b WatchSource:0}: Error finding container 001ce20f153953e3e5340f690fe5fefc372802be3894966e765bc434e97e4f1b: Status 404 returned error can't find the container with id 001ce20f153953e3e5340f690fe5fefc372802be3894966e765bc434e97e4f1b Oct 30 00:16:09.718270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:09.718187 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" event={"ID":"a4deb7d9-4a19-418f-a8c3-9d3064257bfd","Type":"ContainerStarted","Data":"a91bd721b0a879abf6d31b70e2c49fad988c636c6abd39484f88591a950bc57e"} Oct 30 00:16:09.718270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:09.718233 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" event={"ID":"a4deb7d9-4a19-418f-a8c3-9d3064257bfd","Type":"ContainerStarted","Data":"001ce20f153953e3e5340f690fe5fefc372802be3894966e765bc434e97e4f1b"} Oct 30 00:16:09.763218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:09.763155 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podStartSLOduration=1.763136078 podStartE2EDuration="1.763136078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:09.761969511 +0000 UTC m=+678.696038047" watchObservedRunningTime="2026-10-30 00:16:09.763136078 +0000 UTC m=+678.697204615" Oct 30 00:16:10.010616 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:10.010511 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:10.605462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:10.605428 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:12.178906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:12.178875 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:13.676288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:13.676258 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:15.180781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:15.180734 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:16.249719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.249683 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9"] Oct 30 00:16:16.253203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.253184 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.270825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.270796 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9"] Oct 30 00:16:16.281447 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.281420 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"openshift-service-ca.crt\"" Oct 30 00:16:16.281560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.281456 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"kube-root-ca.crt\"" Oct 30 00:16:16.283789 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.283766 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"simulator-w3-564c8f-kserve-self-signed-certs\"" Oct 30 00:16:16.283880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.283765 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"default-dockercfg-jqzgw\"" Oct 30 00:16:16.307559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.307525 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:16.368043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.368008 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-tmp-dir\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.368043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.368045 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10585d8e-dac6-4873-a39c-86ca0525f1b6-tls-certs\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.368298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.368075 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-dshm\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.368298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.368211 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-home\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.368298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.368265 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bxh\" (UniqueName: \"kubernetes.io/projected/10585d8e-dac6-4873-a39c-86ca0525f1b6-kube-api-access-r5bxh\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.368408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.368354 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-model-cache\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470017 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-tmp-dir\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470056 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10585d8e-dac6-4873-a39c-86ca0525f1b6-tls-certs\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470082 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-dshm\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470111 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-home\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470137 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bxh\" (UniqueName: \"kubernetes.io/projected/10585d8e-dac6-4873-a39c-86ca0525f1b6-kube-api-access-r5bxh\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470169 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-model-cache\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470498 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-home\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470511 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-tmp-dir\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.470654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.470634 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-model-cache\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.472491 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.472467 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-dshm\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.472787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.472768 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10585d8e-dac6-4873-a39c-86ca0525f1b6-tls-certs\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.494290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.494261 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bxh\" (UniqueName: \"kubernetes.io/projected/10585d8e-dac6-4873-a39c-86ca0525f1b6-kube-api-access-r5bxh\") pod \"simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.566035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.566000 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:16.687041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.687008 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:16.882494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:16.882336 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9"] Oct 30 00:16:17.769057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:17.769021 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" event={"ID":"10585d8e-dac6-4873-a39c-86ca0525f1b6","Type":"ContainerStarted","Data":"9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937"} Oct 30 00:16:17.769436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:17.769067 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" event={"ID":"10585d8e-dac6-4873-a39c-86ca0525f1b6","Type":"ContainerStarted","Data":"b31d838109659e65b5d8f745f1d44a28ff0c8386e1b2eb50e894a2242a8037c4"} Oct 30 00:16:17.812503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:17.812451 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podStartSLOduration=1.812432717 podStartE2EDuration="1.812432717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:17.810760668 +0000 UTC m=+686.744829204" watchObservedRunningTime="2026-10-30 00:16:17.812432717 +0000 UTC m=+686.746501254" Oct 30 00:16:18.275001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:18.274969 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:18.566640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:18.566565 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:19.009991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:19.009901 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:19.024471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:19.024440 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:19.789809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:19.789757 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:19.791428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:19.791408 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:16:19.949658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:19.949618 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:20.057746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:20.057715 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:21.012825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:21.012791 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:21.296834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:21.296753 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:22.888932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:22.888892 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:24.537392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:24.537327 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:26.003049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:26.003017 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:26.566825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:26.566792 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:26.580057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:26.580023 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:26.830538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:26.830452 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:16:27.613035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:27.612983 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:29.185169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:29.185133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:30.676911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:30.676878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:31.666601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.666542 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq"] Oct 30 00:16:31.670150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.670127 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.737035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.737004 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"premium-w4-02665b-kserve-self-signed-certs\"" Oct 30 00:16:31.737035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.737030 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"default-dockercfg-tksv4\"" Oct 30 00:16:31.737473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.737273 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"openshift-service-ca.crt\"" Oct 30 00:16:31.737473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.737311 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"kube-root-ca.crt\"" Oct 30 00:16:31.739908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.739888 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq"] Oct 30 00:16:31.741977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.741946 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tls-certs\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.742115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.742074 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tmp-dir\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.742184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.742147 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-home\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.742240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.742202 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-model-cache\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.742240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.742217 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdmj\" (UniqueName: \"kubernetes.io/projected/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-kube-api-access-2jdmj\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.742351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.742330 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-dshm\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.838637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.838604 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:31.843610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.843550 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-home\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.843792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.843670 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-model-cache\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.843792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.843698 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdmj\" (UniqueName: \"kubernetes.io/projected/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-kube-api-access-2jdmj\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.843792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.843764 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-dshm\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.843971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.843792 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tls-certs\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.843971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.843902 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tmp-dir\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.844084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.844004 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-home\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.844084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.844051 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-model-cache\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.844224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.844204 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tmp-dir\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.846127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.846104 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-dshm\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.846263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.846245 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tls-certs\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.867879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.867855 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdmj\" (UniqueName: \"kubernetes.io/projected/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-kube-api-access-2jdmj\") pod \"premium-w4-02665b-kserve-6dd9d6d6f-2z2dq\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:31.985894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:31.985799 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:32.287144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:32.287111 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq"] Oct 30 00:16:32.288600 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:16:32.288539 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfb6c7a9_f3dd_4f0d_8081_d274df193a9d.slice/crio-3e4968eeb638532cc0613fe19793ca4daac82de50973c515642f9845eb730d72 WatchSource:0}: Error finding container 3e4968eeb638532cc0613fe19793ca4daac82de50973c515642f9845eb730d72: Status 404 returned error can't find the container with id 3e4968eeb638532cc0613fe19793ca4daac82de50973c515642f9845eb730d72 Oct 30 00:16:32.290187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:32.290159 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:32.854164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:32.854133 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" event={"ID":"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d","Type":"ContainerStarted","Data":"1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba"} Oct 30 00:16:32.854523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:32.854174 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" event={"ID":"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d","Type":"ContainerStarted","Data":"3e4968eeb638532cc0613fe19793ca4daac82de50973c515642f9845eb730d72"} Oct 30 00:16:32.898540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:32.898490 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podStartSLOduration=1.898476616 podStartE2EDuration="1.898476616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:32.896064211 +0000 UTC m=+701.830132745" watchObservedRunningTime="2026-10-30 00:16:32.898476616 +0000 UTC m=+701.832545150" Oct 30 00:16:32.985893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:32.985856 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:33.785192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:33.785155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:35.372327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:35.372295 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:36.454006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:36.453973 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:36.975724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:36.975692 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:37.102515 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:16:37.102478 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:16:38.462741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:38.462699 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:40.071524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:40.071485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:41.591867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:41.591830 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:41.986283 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:41.986204 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:41.999249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:41.999224 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:42.708330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:42.708296 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:42.817331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:42.817294 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:42.922342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:42.922304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:42.923078 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:42.923053 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:16:43.167825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:43.167790 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:44.691064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:44.691036 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:46.188082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:46.188043 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:47.785380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:47.785339 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:49.384779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:49.384731 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:50.879850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:50.879811 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:51.906099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:51.906062 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:52.519186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:52.519134 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:54.151906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:54.151865 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:55.584342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:55.584307 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:55.690531 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:55.690493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:57.066648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:57.066619 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:58.574309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.574281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:16:58.771564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.771525 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj"] Oct 30 00:16:58.776221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.776197 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.796047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.796020 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6c45bb-openshift-default-dockercfg-d5ghz\"" Oct 30 00:16:58.812906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.812876 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj"] Oct 30 00:16:58.948150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948057 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c90b78a-c0c8-43ee-b682-097031cf6734-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948104 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948123 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9trf\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-kube-api-access-c9trf\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948254 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948290 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948323 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948409 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948450 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:58.948798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:58.948517 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049327 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049367 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9trf\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-kube-api-access-c9trf\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049496 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049557 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049624 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049663 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049689 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049720 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049803 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049928 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.049978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.049964 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c90b78a-c0c8-43ee-b682-097031cf6734-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.050173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.050151 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.050350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.050335 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.050669 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.050641 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c90b78a-c0c8-43ee-b682-097031cf6734-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.051913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.051893 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.052302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.052280 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.073290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.073258 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.073387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.073369 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9trf\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-kube-api-access-c9trf\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.089812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.089791 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:16:59.405286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.405248 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj"] Oct 30 00:16:59.406715 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:16:59.406690 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c90b78a_c0c8_43ee_b682_097031cf6734.slice/crio-00dc8a5d18606ba533bbe1d6c78af7365bd5430545865b760ab8ac98144b015b WatchSource:0}: Error finding container 00dc8a5d18606ba533bbe1d6c78af7365bd5430545865b760ab8ac98144b015b: Status 404 returned error can't find the container with id 00dc8a5d18606ba533bbe1d6c78af7365bd5430545865b760ab8ac98144b015b Oct 30 00:16:59.409025 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.408993 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:16:59.409114 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.409069 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:16:59.409170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:16:59.409116 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:00.015554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:00.015502 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" event={"ID":"1c90b78a-c0c8-43ee-b682-097031cf6734","Type":"ContainerStarted","Data":"9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741"} Oct 30 00:17:00.015944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:00.015627 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" event={"ID":"1c90b78a-c0c8-43ee-b682-097031cf6734","Type":"ContainerStarted","Data":"00dc8a5d18606ba533bbe1d6c78af7365bd5430545865b760ab8ac98144b015b"} Oct 30 00:17:00.080988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:00.080927 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podStartSLOduration=2.08090798 podStartE2EDuration="2.08090798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:00.077935044 +0000 UTC m=+729.012003604" watchObservedRunningTime="2026-10-30 00:17:00.08090798 +0000 UTC m=+729.014976518" Oct 30 00:17:00.090916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:00.090883 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:17:00.092502 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:00.092482 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:17:00.092623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:00.092531 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 00:17:00.178321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:00.178284 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:01.090438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:01.090407 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:17:01.090878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:01.090466 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 00:17:01.705164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:01.705126 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:02.090448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:02.090409 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:17:02.090885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:02.090479 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 00:17:03.091149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.091114 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:17:03.091566 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.091183 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 00:17:03.314270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.314220 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:03.315637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.315610 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7"] Oct 30 00:17:03.318806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.318778 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.336211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.336146 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"openshift-service-ca.crt\"" Oct 30 00:17:03.349517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.349426 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"e2e-conflict-a-cee02204-kserve-self-signed-certs\"" Oct 30 00:17:03.349517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.349442 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"kube-root-ca.crt\"" Oct 30 00:17:03.349517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.349470 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"default-dockercfg-s9svd\"" Oct 30 00:17:03.378117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.377878 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7"] Oct 30 00:17:03.407023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.406979 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81abc5-4bea-48b9-bd03-02012f98883a-tls-certs\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.407190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.407081 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-dshm\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.407190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.407115 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-tmp-dir\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.407306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.407190 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6s8s\" (UniqueName: \"kubernetes.io/projected/9f81abc5-4bea-48b9-bd03-02012f98883a-kube-api-access-t6s8s\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.407306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.407289 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-home\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.407398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.407320 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-model-cache\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.419243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.419208 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:03.508175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508139 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-model-cache\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.508383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508241 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81abc5-4bea-48b9-bd03-02012f98883a-tls-certs\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.508383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508315 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-dshm\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.508383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508357 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-tmp-dir\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.508565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508392 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6s8s\" (UniqueName: \"kubernetes.io/projected/9f81abc5-4bea-48b9-bd03-02012f98883a-kube-api-access-t6s8s\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.508565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508513 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-home\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.508738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508643 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-model-cache\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.508913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.508891 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-home\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.509161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.509142 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-tmp-dir\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.512952 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.512898 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81abc5-4bea-48b9-bd03-02012f98883a-tls-certs\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.513762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.513733 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-dshm\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.538954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.538922 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6s8s\" (UniqueName: \"kubernetes.io/projected/9f81abc5-4bea-48b9-bd03-02012f98883a-kube-api-access-t6s8s\") pod \"e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.631889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.631812 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:03.991336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:03.991303 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7"] Oct 30 00:17:03.992232 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:17:03.992191 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f81abc5_4bea_48b9_bd03_02012f98883a.slice/crio-72b261248d780464f41c66438cba7999e653a122863f2308b0a97071a4437d80 WatchSource:0}: Error finding container 72b261248d780464f41c66438cba7999e653a122863f2308b0a97071a4437d80: Status 404 returned error can't find the container with id 72b261248d780464f41c66438cba7999e653a122863f2308b0a97071a4437d80 Oct 30 00:17:04.048142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:04.048114 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" event={"ID":"9f81abc5-4bea-48b9-bd03-02012f98883a","Type":"ContainerStarted","Data":"72b261248d780464f41c66438cba7999e653a122863f2308b0a97071a4437d80"} Oct 30 00:17:04.090169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:04.090139 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:17:04.090312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:04.090187 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 00:17:05.078265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:05.078143 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" event={"ID":"9f81abc5-4bea-48b9-bd03-02012f98883a","Type":"ContainerStarted","Data":"4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca"} Oct 30 00:17:05.081192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:05.081164 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:05.091160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:05.091132 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:17:05.091321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:05.091182 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 00:17:05.125862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:05.125807 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" podStartSLOduration=2.12579313 podStartE2EDuration="2.12579313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:05.123129278 +0000 UTC m=+734.057197836" watchObservedRunningTime="2026-10-30 00:17:05.12579313 +0000 UTC m=+734.059861665" Oct 30 00:17:05.632765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:05.632731 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:06.160658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:06.160625 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:17:06.161118 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:06.160930 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:17:06.161780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:06.161764 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:17:06.775270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:06.775233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:07.099969 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:17:07.099917 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:17:08.271837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.271801 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:08.763065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.763018 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth"] Oct 30 00:17:08.766913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.766883 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.788382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.788348 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth"] Oct 30 00:17:08.797411 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.797386 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"distinct-w6-6c45bb-kserve-self-signed-certs\"" Oct 30 00:17:08.817065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.817038 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:08.898011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.897974 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tls-certs\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.898187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.898035 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-dshm\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.898187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.898124 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-home\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.898319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.898245 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tmp-dir\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.898319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.898310 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qdc\" (UniqueName: \"kubernetes.io/projected/871d04cb-19f3-4fc4-abff-ee5f3483da6b-kube-api-access-f6qdc\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.898392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.898375 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-model-cache\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.999548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.999514 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-model-cache\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.999548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.999552 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tls-certs\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.999812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.999599 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-dshm\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.999812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.999644 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-home\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.999812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.999750 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tmp-dir\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:08.999991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:08.999841 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qdc\" (UniqueName: \"kubernetes.io/projected/871d04cb-19f3-4fc4-abff-ee5f3483da6b-kube-api-access-f6qdc\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.000052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.000015 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-model-cache\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.000129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.000101 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-home\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.000217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.000197 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tmp-dir\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.001962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.001932 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-dshm\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.002131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.002112 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tls-certs\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.024706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.024626 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qdc\" (UniqueName: \"kubernetes.io/projected/871d04cb-19f3-4fc4-abff-ee5f3483da6b-kube-api-access-f6qdc\") pod \"distinct-w6-6c45bb-kserve-648b9f4887-hcxth\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.081636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.081605 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:09.386687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.386643 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth"] Oct 30 00:17:09.388303 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:17:09.388278 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod871d04cb_19f3_4fc4_abff_ee5f3483da6b.slice/crio-2f53dc4827f035345a4da56ea0df3aab92d7cd4e2aee3d4ef297d3add394f290 WatchSource:0}: Error finding container 2f53dc4827f035345a4da56ea0df3aab92d7cd4e2aee3d4ef297d3add394f290: Status 404 returned error can't find the container with id 2f53dc4827f035345a4da56ea0df3aab92d7cd4e2aee3d4ef297d3add394f290 Oct 30 00:17:09.946921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:09.946882 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:10.109648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:10.109614 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" event={"ID":"871d04cb-19f3-4fc4-abff-ee5f3483da6b","Type":"ContainerStarted","Data":"615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946"} Oct 30 00:17:10.109803 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:10.109656 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" event={"ID":"871d04cb-19f3-4fc4-abff-ee5f3483da6b","Type":"ContainerStarted","Data":"2f53dc4827f035345a4da56ea0df3aab92d7cd4e2aee3d4ef297d3add394f290"} Oct 30 00:17:10.153096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:10.153040 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podStartSLOduration=2.153025349 podStartE2EDuration="2.153025349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:10.151712935 +0000 UTC m=+739.085781480" watchObservedRunningTime="2026-10-30 00:17:10.153025349 +0000 UTC m=+739.087093883" Oct 30 00:17:11.082626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:11.082593 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:11.470342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:11.470259 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:13.075084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:13.075050 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:13.632525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:13.632493 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:13.645244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:13.645217 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:14.143471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:14.143440 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:17:14.673707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:14.673673 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:16.177223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:16.177181 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:17.778047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:17.778013 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:19.082309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:19.082277 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:19.094715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:19.094690 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:19.179855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:19.179821 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:17:19.264977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:19.264925 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:19.821478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:19.821428 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:20.873073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:20.873037 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:22.378259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:22.378222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:23.958898 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:23.958864 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:25.043944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.043906 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng"] Oct 30 00:17:25.047479 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.047450 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.071059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.071031 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng"] Oct 30 00:17:25.077619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.077595 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-564c8f-openshift-default-dockercfg-f7gl6\"" Oct 30 00:17:25.097098 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097059 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097206 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097238 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrgx\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-kube-api-access-hqrgx\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097340 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097370 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097513 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097597 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097689 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.097766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.097742 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.198905 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.198872 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.198905 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.198911 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrgx\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-kube-api-access-hqrgx\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.198955 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.198971 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199024 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199063 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199112 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199144 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199201 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199669 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199705 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199713 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199764 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.199906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.199797 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.201450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.201431 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.201562 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.201544 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.222775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.222745 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.222985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.222966 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrgx\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-kube-api-access-hqrgx\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.360145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.360103 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:25.481545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.481514 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:25.678567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.678541 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng"] Oct 30 00:17:25.679626 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:17:25.679600 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2186b19c_57bf_41cb_bf4e_a9800a8e2934.slice/crio-893f673bf8ea922b3ae3c578fbdab39fb2ef21ebf595873732c14605d9814208 WatchSource:0}: Error finding container 893f673bf8ea922b3ae3c578fbdab39fb2ef21ebf595873732c14605d9814208: Status 404 returned error can't find the container with id 893f673bf8ea922b3ae3c578fbdab39fb2ef21ebf595873732c14605d9814208 Oct 30 00:17:25.681982 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.681952 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:25.682039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.682015 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:25.682077 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:25.682044 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:26.199728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:26.199694 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" event={"ID":"2186b19c-57bf-41cb-bf4e-a9800a8e2934","Type":"ContainerStarted","Data":"05c0b6e780b638933a6a60865cded6bb280532686d74a304aaed87f216b0134e"} Oct 30 00:17:26.200141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:26.199748 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" event={"ID":"2186b19c-57bf-41cb-bf4e-a9800a8e2934","Type":"ContainerStarted","Data":"893f673bf8ea922b3ae3c578fbdab39fb2ef21ebf595873732c14605d9814208"} Oct 30 00:17:26.368677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:26.368640 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:26.376796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:26.376761 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:17:26.377000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:26.376822 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 00:17:27.179641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:27.179600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:27.364596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:27.362946 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:17:27.364596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:27.363004 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 00:17:28.367599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:28.362934 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:17:28.367599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:28.362996 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 00:17:28.622845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:28.622714 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:29.370597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:29.367934 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:17:29.370597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:29.367999 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 00:17:30.118202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:30.118165 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:30.372051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:30.371954 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:17:30.372639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:30.372607 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 00:17:31.375599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:31.372957 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:17:31.375599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:31.373025 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 00:17:31.983441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:31.983393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:32.515650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:32.515615 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:32.516363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:32.516339 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:32.517768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:32.517740 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:17:32.576826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:32.576763 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podStartSLOduration=7.576744298 podStartE2EDuration="7.576744298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:26.277030056 +0000 UTC m=+755.211098589" watchObservedRunningTime="2026-10-30 00:17:32.576744298 +0000 UTC m=+761.510812835" Oct 30 00:17:33.490568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.490534 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:33.828549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.828516 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p"] Oct 30 00:17:33.833187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.833167 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.852031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.852005 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-02665b-openshift-default-dockercfg-c72fk\"" Oct 30 00:17:33.870367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.870334 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p"] Oct 30 00:17:33.939915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.939884 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzkn\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-kube-api-access-pwzkn\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940003 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940059 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940169 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f1b969-18ae-48bf-bc75-b859ab445850-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940283 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940337 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940360 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940398 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:33.940456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:33.940440 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041291 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f1b969-18ae-48bf-bc75-b859ab445850-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041413 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041446 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041494 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041549 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041590 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041652 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzkn\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-kube-api-access-pwzkn\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041741 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041819 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.041966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.041829 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.042192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.042168 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f1b969-18ae-48bf-bc75-b859ab445850-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.042192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.042173 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.042309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.042213 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.042558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.042525 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.044761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.044739 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.044844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.044813 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.066032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.066012 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzkn\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-kube-api-access-pwzkn\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.066134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.066114 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.146743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.146658 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:34.472353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.472321 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p"] Oct 30 00:17:34.474156 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:17:34.474123 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f1b969_18ae_48bf_bc75_b859ab445850.slice/crio-62f6f31c8ab10fa73d4d53f262381ee003ca07cdebe80cbae4dd3ce60d5869cb WatchSource:0}: Error finding container 62f6f31c8ab10fa73d4d53f262381ee003ca07cdebe80cbae4dd3ce60d5869cb: Status 404 returned error can't find the container with id 62f6f31c8ab10fa73d4d53f262381ee003ca07cdebe80cbae4dd3ce60d5869cb Oct 30 00:17:34.476524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.476491 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:34.476666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.476609 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:34.476666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:34.476660 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:35.082526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:35.082464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:35.279644 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:35.279597 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" event={"ID":"d5f1b969-18ae-48bf-bc75-b859ab445850","Type":"ContainerStarted","Data":"1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd"} Oct 30 00:17:35.279793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:35.279677 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" event={"ID":"d5f1b969-18ae-48bf-bc75-b859ab445850","Type":"ContainerStarted","Data":"62f6f31c8ab10fa73d4d53f262381ee003ca07cdebe80cbae4dd3ce60d5869cb"} Oct 30 00:17:35.345936 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:35.345835 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" podStartSLOduration=2.34582073 podStartE2EDuration="2.34582073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:35.34513403 +0000 UTC m=+764.279202593" watchObservedRunningTime="2026-10-30 00:17:35.34582073 +0000 UTC m=+764.279889265" Oct 30 00:17:35.808502 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:35.808471 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:36.147659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:36.147554 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:36.149358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:36.149332 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:17:36.149492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:36.149378 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 00:17:36.583261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:36.583207 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:36.801347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:36.801296 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:37.104552 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:17:37.104516 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:17:37.147184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:37.147149 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:17:37.147380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:37.147226 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 00:17:38.147222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:38.147183 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:17:38.147644 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:38.147252 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 00:17:38.439013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:38.438915 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:39.147702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:39.147669 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:17:39.148190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:39.147720 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 00:17:39.989455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:39.989417 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:40.147266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:40.147231 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:17:40.147443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:40.147283 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 00:17:41.152253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:41.152226 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:41.152726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:41.152689 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:41.153421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:41.153399 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:17:41.477732 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:41.477640 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:43.241958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.239727 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:43.908753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.908719 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx"] Oct 30 00:17:43.913876 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.913847 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.951324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.951278 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx"] Oct 30 00:17:43.981673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.981418 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.982040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.982010 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npgf\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-kube-api-access-4npgf\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.982337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.982317 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.982511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.982492 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.982650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.982634 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c8a69-4712-4df7-a958-7f53f9338811-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.982771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.982756 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.982925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.982909 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.983067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.983051 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c8a69-4712-4df7-a958-7f53f9338811-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:43.983241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:43.983225 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.084451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.084405 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.084662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.084485 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.084662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.084512 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c8a69-4712-4df7-a958-7f53f9338811-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.084774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.084730 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.084851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.084819 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.084954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.084929 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c8a69-4712-4df7-a958-7f53f9338811-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.085051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.084999 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.085051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.085038 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.085157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.085047 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.085157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.085146 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npgf\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-kube-api-access-4npgf\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.085437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.085410 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c8a69-4712-4df7-a958-7f53f9338811-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.085682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.085661 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.085927 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.085903 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.086132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.086108 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.088414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.087802 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.088414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.088051 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c8a69-4712-4df7-a958-7f53f9338811-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.109303 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.109270 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.109520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.109490 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npgf\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-kube-api-access-4npgf\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.244816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.244726 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:44.570781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.570752 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx"] Oct 30 00:17:44.571550 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:17:44.571524 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51c8a69_4712_4df7_a958_7f53f9338811.slice/crio-ec239921e2fcd684d2b62e1e8ee63298aafef91eb7b6c54ec5f21d843c09f0a2 WatchSource:0}: Error finding container ec239921e2fcd684d2b62e1e8ee63298aafef91eb7b6c54ec5f21d843c09f0a2: Status 404 returned error can't find the container with id ec239921e2fcd684d2b62e1e8ee63298aafef91eb7b6c54ec5f21d843c09f0a2 Oct 30 00:17:44.573994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.573960 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:44.574069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.574026 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:44.574069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.574054 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:17:44.761827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:44.761776 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:45.346058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:45.346019 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" event={"ID":"c51c8a69-4712-4df7-a958-7f53f9338811","Type":"ContainerStarted","Data":"9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b"} Oct 30 00:17:45.346461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:45.346078 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" event={"ID":"c51c8a69-4712-4df7-a958-7f53f9338811","Type":"ContainerStarted","Data":"ec239921e2fcd684d2b62e1e8ee63298aafef91eb7b6c54ec5f21d843c09f0a2"} Oct 30 00:17:45.419865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:45.419804 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podStartSLOduration=2.41978991 podStartE2EDuration="2.41978991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:45.404721604 +0000 UTC m=+774.338790151" watchObservedRunningTime="2026-10-30 00:17:45.41978991 +0000 UTC m=+774.353858512" Oct 30 00:17:46.214411 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:46.214381 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:46.245168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:46.245129 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:46.246971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:46.246946 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:17:46.247050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:46.247010 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 00:17:46.926783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:46.926752 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:47.245967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:47.245883 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:17:47.245967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:47.245943 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 00:17:47.818605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:47.818535 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:48.245469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:48.245387 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:17:48.245469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:48.245448 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 00:17:49.246115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:49.246083 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:17:49.246525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:49.246141 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 00:17:49.404216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:49.404182 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:50.245157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:50.245117 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:17:50.245351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:50.245175 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 00:17:51.010875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:51.010816 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:51.245515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:51.245480 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:17:51.245704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:51.245533 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 00:17:52.250167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:52.250131 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:52.250615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:52.250429 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:52.251194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:52.251174 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:17:52.569896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:52.569861 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:54.094941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:54.094901 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:55.586553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:55.586512 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:57.177813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:57.177762 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:57.287761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:57.287726 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:17:58.787298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:17:58.787264 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:00.271694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:00.271658 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:00.512545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:00.512511 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:01.784211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:01.784169 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:03.036023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:03.035982 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:03.520921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:03.520888 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:05.093181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:05.093130 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:06.548604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:06.548560 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:07.099973 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:18:07.099932 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:18:07.984557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:07.984524 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:09.409808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.409776 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw"] Oct 30 00:18:09.413890 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.413869 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.434384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.434355 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw"] Oct 30 00:18:09.442058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.442034 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-322d20\"/\"distinct-w2-322d20-kserve-self-signed-certs\"" Oct 30 00:18:09.467354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.467326 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tmp-dir\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.467530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.467509 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-dshm\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.467589 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.467551 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7tjp\" (UniqueName: \"kubernetes.io/projected/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-kube-api-access-w7tjp\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.467655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.467635 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-model-cache\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.467770 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.467726 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tls-certs\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.467770 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.467764 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-home\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.474544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.474521 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:09.569606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.569559 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-dshm\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.569783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.569614 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tjp\" (UniqueName: \"kubernetes.io/projected/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-kube-api-access-w7tjp\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.569783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.569639 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-model-cache\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.569783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.569678 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tls-certs\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.569783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.569703 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-home\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.570013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.569790 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tmp-dir\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.570075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.570023 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-model-cache\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.570170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.570145 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tmp-dir\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.570223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.570208 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-home\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.571921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.571897 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-dshm\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.572020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.572002 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tls-certs\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.579433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.579409 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:09.593081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.593055 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tjp\" (UniqueName: \"kubernetes.io/projected/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-kube-api-access-w7tjp\") pod \"distinct-w2-322d20-kserve-c98d94958-58blw\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:09.730825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:09.730741 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:10.034763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:10.034734 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw"] Oct 30 00:18:10.036060 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:18:10.036030 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ffa82da_a0b6_473b_9556_a11cd7e0a0b9.slice/crio-8ccfd0f2a99cc21f3b9fbfb939e8549cc537e5236f6ce2600ed875be7b6e0658 WatchSource:0}: Error finding container 8ccfd0f2a99cc21f3b9fbfb939e8549cc537e5236f6ce2600ed875be7b6e0658: Status 404 returned error can't find the container with id 8ccfd0f2a99cc21f3b9fbfb939e8549cc537e5236f6ce2600ed875be7b6e0658 Oct 30 00:18:10.530292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:10.530255 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" event={"ID":"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9","Type":"ContainerStarted","Data":"fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f"} Oct 30 00:18:10.530292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:10.530296 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" event={"ID":"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9","Type":"ContainerStarted","Data":"8ccfd0f2a99cc21f3b9fbfb939e8549cc537e5236f6ce2600ed875be7b6e0658"} Oct 30 00:18:10.573796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:10.573744 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podStartSLOduration=1.573730533 podStartE2EDuration="1.573730533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:10.571965685 +0000 UTC m=+799.506034227" watchObservedRunningTime="2026-10-30 00:18:10.573730533 +0000 UTC m=+799.507799112" Oct 30 00:18:10.731020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:10.730973 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:10.995755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:10.995725 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:12.467888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:12.467853 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:14.108962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:14.108920 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:15.967134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:15.967090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:17.495072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:17.495035 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:18.990597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:18.990528 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:19.731822 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:19.731784 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:19.744991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:19.744955 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:20.484398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:20.484361 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:20.610460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:20.610426 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:18:21.983420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:21.983380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:23.590866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:23.590817 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:24.996396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:24.996350 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:26.413790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:26.413748 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:28.046143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.046107 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb"] Oct 30 00:18:28.051466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.051443 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.068186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.068152 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb"] Oct 30 00:18:28.068850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.068787 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:28.076270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.076221 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"distinct-2-w4-02665b-kserve-self-signed-certs\"" Oct 30 00:18:28.127462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.127423 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-dshm\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.127462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.127467 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-tmp-dir\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.127704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.127530 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f65f05-73d3-4d40-81f7-3300f86869ef-tls-certs\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.127704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.127563 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-home\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.127794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.127743 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-model-cache\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.127794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.127772 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbkd\" (UniqueName: \"kubernetes.io/projected/89f65f05-73d3-4d40-81f7-3300f86869ef-kube-api-access-fjbkd\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.181000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.180960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:28.228443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.228406 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-dshm\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.228667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.228449 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-tmp-dir\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.228667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.228489 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f65f05-73d3-4d40-81f7-3300f86869ef-tls-certs\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.228667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.228558 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-home\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.228844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.228787 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-model-cache\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.228844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.228816 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbkd\" (UniqueName: \"kubernetes.io/projected/89f65f05-73d3-4d40-81f7-3300f86869ef-kube-api-access-fjbkd\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.229018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.228989 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-tmp-dir\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.229172 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.229148 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-home\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.229292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.229274 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-model-cache\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.230741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.230722 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-dshm\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.231027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.231009 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f65f05-73d3-4d40-81f7-3300f86869ef-tls-certs\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.253558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.253524 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbkd\" (UniqueName: \"kubernetes.io/projected/89f65f05-73d3-4d40-81f7-3300f86869ef-kube-api-access-fjbkd\") pod \"distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.372008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.371975 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:28.680821 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:28.680794 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb"] Oct 30 00:18:28.682234 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:18:28.682207 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f65f05_73d3_4d40_81f7_3300f86869ef.slice/crio-7f9a58e8b31dee5cf4826e14750b82b047320163090e88eef5d81e47ea1c7361 WatchSource:0}: Error finding container 7f9a58e8b31dee5cf4826e14750b82b047320163090e88eef5d81e47ea1c7361: Status 404 returned error can't find the container with id 7f9a58e8b31dee5cf4826e14750b82b047320163090e88eef5d81e47ea1c7361 Oct 30 00:18:29.655741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:29.655705 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" event={"ID":"89f65f05-73d3-4d40-81f7-3300f86869ef","Type":"ContainerStarted","Data":"d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b"} Oct 30 00:18:29.656225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:29.655756 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" event={"ID":"89f65f05-73d3-4d40-81f7-3300f86869ef","Type":"ContainerStarted","Data":"7f9a58e8b31dee5cf4826e14750b82b047320163090e88eef5d81e47ea1c7361"} Oct 30 00:18:29.671867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:29.671833 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:29.700379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:29.700335 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podStartSLOduration=1.7003231749999999 podStartE2EDuration="1.700323175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:29.697651635 +0000 UTC m=+818.631720187" watchObservedRunningTime="2026-10-30 00:18:29.700323175 +0000 UTC m=+818.634391709" Oct 30 00:18:30.372852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:30.372816 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:31.139001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:31.138962 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:32.657292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:32.657246 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:32.979293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:32.979194 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:34.171377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:34.171336 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:35.701193 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:35.701158 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:36.713080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:36.713038 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:37.103560 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:18:37.103516 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:18:37.105160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.105132 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp"] Oct 30 00:18:37.109686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.109659 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.129609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.129491 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp"] Oct 30 00:18:37.196832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.196801 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:37.240110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240073 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240189 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240294 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240328 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240354 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240503 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6nmk\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-kube-api-access-x6nmk\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240560 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240697 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.240765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.240726 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.341984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.341946 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.341984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.341985 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342054 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342091 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342110 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342135 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342168 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342264 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6nmk\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-kube-api-access-x6nmk\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342327 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342439 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342554 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342674 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.342803 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.342754 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.343055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.343037 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.344653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.344623 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.344791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.344776 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.366045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.365984 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.366149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.366068 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6nmk\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-kube-api-access-x6nmk\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.426062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.426039 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:37.747598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.747551 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp"] Oct 30 00:18:37.748533 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:18:37.748507 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cc8f76_f6ed_40a9_9d3a_29e03c29fd86.slice/crio-80e23255bac36c0da34a89b275155b0d911943a6134c7e563c8fad1ce7060ca7 WatchSource:0}: Error finding container 80e23255bac36c0da34a89b275155b0d911943a6134c7e563c8fad1ce7060ca7: Status 404 returned error can't find the container with id 80e23255bac36c0da34a89b275155b0d911943a6134c7e563c8fad1ce7060ca7 Oct 30 00:18:37.750930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.750896 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:37.751012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.750965 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:37.751012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:37.750992 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:38.020007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.018690 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:38.378543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.377278 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:38.391311 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.391285 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:38.714780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.714689 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" event={"ID":"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86","Type":"ContainerStarted","Data":"f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43"} Oct 30 00:18:38.714780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.714749 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" event={"ID":"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86","Type":"ContainerStarted","Data":"80e23255bac36c0da34a89b275155b0d911943a6134c7e563c8fad1ce7060ca7"} Oct 30 00:18:38.726379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.726355 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:18:38.823141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.823090 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podStartSLOduration=1.823076575 podStartE2EDuration="1.823076575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:38.77796606 +0000 UTC m=+827.712034594" watchObservedRunningTime="2026-10-30 00:18:38.823076575 +0000 UTC m=+827.757145114" Oct 30 00:18:38.902270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:38.902173 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:39.426897 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:39.426864 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:39.428231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:39.428206 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 00:18:39.428364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:39.428258 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 00:18:40.427053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:40.427015 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 00:18:40.427409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:40.427077 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 00:18:40.576596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:40.576540 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:40.684652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:40.684116 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:41.426494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:41.426460 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 00:18:41.426710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:41.426519 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 00:18:42.210425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:42.210389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:42.426833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:42.426799 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 00:18:42.427029 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:42.426865 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 00:18:43.426722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:43.426687 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 00:18:43.427226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:43.426747 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 00:18:43.798447 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:43.798351 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:44.059380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.059349 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57"] Oct 30 00:18:44.063781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.063760 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.094093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.094060 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57"] Oct 30 00:18:44.151397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151361 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151428 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151518 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151554 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151649 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151726 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8fl\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-kube-api-access-6c8fl\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151769 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151791 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.151910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.151843 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253348 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8fl\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-kube-api-access-6c8fl\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253399 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253517 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253568 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253671 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253755 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253830 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253855 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253866 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.253992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.253937 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.254138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.254109 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.254274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.254248 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.254345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.254305 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.254601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.254559 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.256059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.256040 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.256662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.256641 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.277196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.277171 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.277329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.277309 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8fl\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-kube-api-access-6c8fl\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.378809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.378730 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:44.428143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.428104 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 00:18:44.428645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.428606 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 00:18:44.709741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.709642 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57"] Oct 30 00:18:44.720702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.720624 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:44.720825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.720707 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:44.720825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.720743 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:44.753534 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:44.753499 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" event={"ID":"64732b06-4a87-41c7-b532-cbf9ca01a0b4","Type":"ContainerStarted","Data":"a2b65e343a0eba5e29b30b4954b8cadff15862aa5882247cafd508ae592bd1a6"} Oct 30 00:18:45.370377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:45.370340 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:45.433015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:45.432985 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:45.434837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:45.433817 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:45.434837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:45.434510 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:18:45.768714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:45.768623 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" event={"ID":"64732b06-4a87-41c7-b532-cbf9ca01a0b4","Type":"ContainerStarted","Data":"16b2cfd916aca20744e2346a53c41388dc2242303409953a34489a640e620544"} Oct 30 00:18:45.818353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:45.818307 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podStartSLOduration=1.81829418 podStartE2EDuration="1.81829418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:45.816877795 +0000 UTC m=+834.750946325" watchObservedRunningTime="2026-10-30 00:18:45.81829418 +0000 UTC m=+834.752362714" Oct 30 00:18:46.379043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:46.379004 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:46.380669 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:46.380641 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:18:46.380825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:46.380690 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:18:46.984296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:46.984255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:47.379628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:47.379597 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:18:47.379832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:47.379645 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:18:48.379625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:48.379556 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:18:48.380062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:48.379681 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:18:48.479007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:48.478946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:49.380118 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:49.380076 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:18:49.380607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:49.380140 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:18:49.900594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:49.897474 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7"] Oct 30 00:18:49.931566 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:49.931523 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:50.111807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:50.111771 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:50.380166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:50.380126 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:18:50.380603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:50.380194 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:18:51.379999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:51.379962 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:18:51.380274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:51.380244 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:18:52.141153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.141119 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj"] Oct 30 00:18:52.146787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.146762 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.203601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.198614 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj"] Oct 30 00:18:52.339410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.339185 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.339635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.339456 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.339726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.339482 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.339794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.339737 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.339794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.339786 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.339995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.339980 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78a5d9-5483-4835-b906-6c13fb96461c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.340129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.340115 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.340379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.340362 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.340454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.340442 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47t7\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-kube-api-access-v47t7\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.383597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.382676 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:18:52.383597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.382751 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:18:52.433093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.432926 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:52.446711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.446660 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.446925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.446744 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47t7\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-kube-api-access-v47t7\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.446925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.446881 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.447090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.446932 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.447090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.446962 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.447090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.447008 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.447090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.447065 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.447299 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.447095 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78a5d9-5483-4835-b906-6c13fb96461c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.447299 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.447225 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.448790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.448763 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.449215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.449186 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.449342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.449320 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.456650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.453313 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78a5d9-5483-4835-b906-6c13fb96461c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.456821 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.453778 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.471227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.471195 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.511606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.508422 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.526619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.521280 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47t7\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-kube-api-access-v47t7\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.532838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.532806 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:52.807203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:52.806862 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:53.218733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.218699 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj"] Oct 30 00:18:53.221882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.221845 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:53.221978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.221913 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:53.221978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.221960 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:18:53.437625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.437594 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:53.438150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.437976 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:53.438933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.438912 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:18:53.860001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.859965 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" event={"ID":"8b78a5d9-5483-4835-b906-6c13fb96461c","Type":"ContainerStarted","Data":"beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f"} Oct 30 00:18:53.860255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.860235 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" event={"ID":"8b78a5d9-5483-4835-b906-6c13fb96461c","Type":"ContainerStarted","Data":"9a6215a31020b4ae35ac24f3d16d84fd65cc9494be9fc3c2c5d996158bd8a846"} Oct 30 00:18:53.918375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:53.918329 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podStartSLOduration=1.918315671 podStartE2EDuration="1.918315671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:53.915013732 +0000 UTC m=+842.849082271" watchObservedRunningTime="2026-10-30 00:18:53.918315671 +0000 UTC m=+842.852384206" Oct 30 00:18:54.202024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:54.201923 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:54.808481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:54.808446 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:18:54.809638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:54.809608 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:18:54.809781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:54.809665 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:18:55.808305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:55.808266 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:18:55.808599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:55.808362 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:18:55.909965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:55.909929 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:56.808698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:56.808664 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:18:56.809109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:56.808727 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:18:57.609488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:57.609446 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:57.808721 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:57.808689 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:18:57.809121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:57.808739 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:18:58.013737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.013644 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84"] Oct 30 00:18:58.019094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.019066 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.034251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.034225 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84"] Oct 30 00:18:58.045157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.045124 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"distinct-2-w3-564c8f-kserve-self-signed-certs\"" Oct 30 00:18:58.049391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.049358 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377874c-82d4-452e-af77-fdfc2a90fc76-tls-certs\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.049643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.049521 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-model-cache\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.049643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.049549 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-tmp-dir\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.049643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.049623 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-home\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.049824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.049804 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-dshm\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.049871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.049837 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7cc\" (UniqueName: \"kubernetes.io/projected/d377874c-82d4-452e-af77-fdfc2a90fc76-kube-api-access-dj7cc\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.082488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.082451 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:58.150832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.150773 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-dshm\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.150832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.150846 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7cc\" (UniqueName: \"kubernetes.io/projected/d377874c-82d4-452e-af77-fdfc2a90fc76-kube-api-access-dj7cc\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.151103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.150893 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377874c-82d4-452e-af77-fdfc2a90fc76-tls-certs\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.151103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.151054 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-model-cache\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.151103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.151082 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-tmp-dir\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.151249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.151157 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-home\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.151396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.151372 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-model-cache\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.151483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.151414 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-tmp-dir\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.175597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.151619 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-home\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.175597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.172296 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-dshm\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.181634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.178223 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377874c-82d4-452e-af77-fdfc2a90fc76-tls-certs\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.181634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.181054 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7cc\" (UniqueName: \"kubernetes.io/projected/d377874c-82d4-452e-af77-fdfc2a90fc76-kube-api-access-dj7cc\") pod \"distinct-2-w3-564c8f-kserve-558b5f487c-5nw84\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.335625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.335558 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:58.711273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.711238 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84"] Oct 30 00:18:58.807992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.807963 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:18:58.808130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.808011 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:18:58.908880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.908839 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" event={"ID":"d377874c-82d4-452e-af77-fdfc2a90fc76","Type":"ContainerStarted","Data":"618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7"} Oct 30 00:18:58.909262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.908892 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" event={"ID":"d377874c-82d4-452e-af77-fdfc2a90fc76","Type":"ContainerStarted","Data":"8b6e8c6bb194bc75fedb465a53b8b74ae4f5c209af9a705ea6b64e83dacf018e"} Oct 30 00:18:58.979659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:58.979525 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podStartSLOduration=1.979506619 podStartE2EDuration="1.979506619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:58.950503451 +0000 UTC m=+847.884572034" watchObservedRunningTime="2026-10-30 00:18:58.979506619 +0000 UTC m=+847.913575153" Oct 30 00:18:59.336425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:59.336378 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:18:59.477337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:59.477299 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:18:59.808191 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:59.808165 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:18:59.808357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:18:59.808227 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:19:00.812506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:00.812474 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:19:00.812999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:00.812976 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:19:00.813558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:00.813538 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:19:01.082374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:01.082280 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:02.584912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:02.584883 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:04.061593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.061554 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:04.170996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.170953 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn"] Oct 30 00:19:04.175406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.175380 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.199292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.199262 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn"] Oct 30 00:19:04.268310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268276 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe97ca-af94-4472-9fd2-87ea421bd627-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268314 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268393 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268434 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268620 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268488 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvxc\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-kube-api-access-zkvxc\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268620 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268561 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268713 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268750 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.268868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.268855 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370688 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370605 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe97ca-af94-4472-9fd2-87ea421bd627-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370688 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370653 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370688 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370688 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370715 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370743 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvxc\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-kube-api-access-zkvxc\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370796 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370885 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.370962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.370929 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.371209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.371036 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.371270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.371198 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.371330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.371315 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.371439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.371416 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.371567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.371415 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe97ca-af94-4472-9fd2-87ea421bd627-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.371567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.371475 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.373123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.373098 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.373340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.373319 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.395757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.395732 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.395872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.395857 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvxc\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-kube-api-access-zkvxc\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.495511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.495480 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:04.816819 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:19:04.816776 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbe97ca_af94_4472_9fd2_87ea421bd627.slice/crio-212a99f10cb96893c11c3684345097b83b543bd10064954104064699aec871f6 WatchSource:0}: Error finding container 212a99f10cb96893c11c3684345097b83b543bd10064954104064699aec871f6: Status 404 returned error can't find the container with id 212a99f10cb96893c11c3684345097b83b543bd10064954104064699aec871f6 Oct 30 00:19:04.818102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.818070 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn"] Oct 30 00:19:04.821148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.821107 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:04.821488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.821194 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:04.821488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.821240 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:04.879547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.879515 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:04.925442 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.925404 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" podUID="9f81abc5-4bea-48b9-bd03-02012f98883a" containerName="main" containerID="cri-o://4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca" gracePeriod=45 Oct 30 00:19:04.950993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.950962 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" event={"ID":"6bbe97ca-af94-4472-9fd2-87ea421bd627","Type":"ContainerStarted","Data":"79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453"} Oct 30 00:19:04.951137 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.951025 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" event={"ID":"6bbe97ca-af94-4472-9fd2-87ea421bd627","Type":"ContainerStarted","Data":"212a99f10cb96893c11c3684345097b83b543bd10064954104064699aec871f6"} Oct 30 00:19:04.998689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:04.998634 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podStartSLOduration=0.998620768 podStartE2EDuration="998.620768ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:04.997190077 +0000 UTC m=+853.931258619" watchObservedRunningTime="2026-10-30 00:19:04.998620768 +0000 UTC m=+853.932689307" Oct 30 00:19:05.231208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.231178 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:19:05.387164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.387065 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f81abc5-4bea-48b9-bd03-02012f98883a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81abc5-4bea-48b9-bd03-02012f98883a-tls-certs\") pod \"9f81abc5-4bea-48b9-bd03-02012f98883a\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " Oct 30 00:19:05.387164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.387138 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-model-cache\") pod \"9f81abc5-4bea-48b9-bd03-02012f98883a\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " Oct 30 00:19:05.387407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.387322 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f81abc5-4bea-48b9-bd03-02012f98883a-kube-api-access-t6s8s\" (UniqueName: \"kubernetes.io/projected/9f81abc5-4bea-48b9-bd03-02012f98883a-kube-api-access-t6s8s\") pod \"9f81abc5-4bea-48b9-bd03-02012f98883a\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " Oct 30 00:19:05.387407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.387360 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-home\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-home\") pod \"9f81abc5-4bea-48b9-bd03-02012f98883a\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " Oct 30 00:19:05.387520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.387434 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-dshm\") pod \"9f81abc5-4bea-48b9-bd03-02012f98883a\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " Oct 30 00:19:05.387520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.387512 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-tmp-dir\") pod \"9f81abc5-4bea-48b9-bd03-02012f98883a\" (UID: \"9f81abc5-4bea-48b9-bd03-02012f98883a\") " Oct 30 00:19:05.388909 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.388534 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-tmp-dir" pod "9f81abc5-4bea-48b9-bd03-02012f98883a" (UID: "9f81abc5-4bea-48b9-bd03-02012f98883a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:19:05.389160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.389128 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-model-cache" pod "9f81abc5-4bea-48b9-bd03-02012f98883a" (UID: "9f81abc5-4bea-48b9-bd03-02012f98883a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:19:05.389358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.389337 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-home" pod "9f81abc5-4bea-48b9-bd03-02012f98883a" (UID: "9f81abc5-4bea-48b9-bd03-02012f98883a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:19:05.392637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.392602 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f81abc5-4bea-48b9-bd03-02012f98883a-tls-certs" pod "9f81abc5-4bea-48b9-bd03-02012f98883a" (UID: "9f81abc5-4bea-48b9-bd03-02012f98883a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:19:05.392637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.392626 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-dshm" pod "9f81abc5-4bea-48b9-bd03-02012f98883a" (UID: "9f81abc5-4bea-48b9-bd03-02012f98883a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:19:05.400691 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.400627 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f81abc5-4bea-48b9-bd03-02012f98883a-kube-api-access-t6s8s" pod "9f81abc5-4bea-48b9-bd03-02012f98883a" (UID: "9f81abc5-4bea-48b9-bd03-02012f98883a"). InnerVolumeSpecName "kube-api-access-t6s8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:19:05.489434 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.489399 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6s8s\" (UniqueName: \"kubernetes.io/projected/9f81abc5-4bea-48b9-bd03-02012f98883a-kube-api-access-t6s8s\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:19:05.489434 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.489427 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:19:05.489434 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.489439 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:19:05.489713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.489449 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:19:05.489713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.489457 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81abc5-4bea-48b9-bd03-02012f98883a-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:19:05.489713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.489465 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81abc5-4bea-48b9-bd03-02012f98883a-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:19:05.496677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.496641 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:05.498227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.498206 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:19:05.498316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.498250 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:19:05.704620 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.704503 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:05.982503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.982406 4433 generic.go:356] "Generic (PLEG): container finished" podID="9f81abc5-4bea-48b9-bd03-02012f98883a" containerID="4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca" exitCode=0 Oct 30 00:19:05.982503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.982472 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" Oct 30 00:19:05.982739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.982488 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" event={"ID":"9f81abc5-4bea-48b9-bd03-02012f98883a","Type":"ContainerDied","Data":"4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca"} Oct 30 00:19:05.982739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.982539 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" event={"ID":"9f81abc5-4bea-48b9-bd03-02012f98883a","Type":"ContainerDied","Data":"72b261248d780464f41c66438cba7999e653a122863f2308b0a97071a4437d80"} Oct 30 00:19:05.982739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.982555 4433 scope.go:122] "RemoveContainer" containerID="4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca" Oct 30 00:19:05.992774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.992756 4433 scope.go:122] "RemoveContainer" containerID="4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca" Oct 30 00:19:05.993055 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:19:05.993031 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca\": container with ID starting with 4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca not found: ID does not exist" containerID="4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca" Oct 30 00:19:05.993171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:05.993066 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca"} err="failed to get container status \"4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca\": rpc error: code = NotFound desc = could not find container \"4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca\": container with ID starting with 4f10a2dbfc5667d60ab4587506aa374bcd32c0dd33ac8a32d017de9f720604ca not found: ID does not exist" Oct 30 00:19:06.036315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:06.036279 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7"] Oct 30 00:19:06.049008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:06.048987 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7"] Oct 30 00:19:06.133082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:06.133044 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/e2e-conflict-a-cee02204-kserve-6689fc8fdd-6w8w7" podUID="9f81abc5-4bea-48b9-bd03-02012f98883a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:06.496611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:06.496550 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:19:06.497120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:06.496647 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:19:07.104772 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:19:07.104735 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:19:07.185837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:07.185799 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f81abc5-4bea-48b9-bd03-02012f98883a" path="/var/lib/kubelet/pods/9f81abc5-4bea-48b9-bd03-02012f98883a/volumes" Oct 30 00:19:07.331480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:07.331452 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:07.496919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:07.496832 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:19:07.496919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:07.496906 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:19:08.335737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.335703 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:19:08.348036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.348012 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:19:08.496373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.496343 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:19:08.496544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.496405 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:19:08.684711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.683720 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z"] Oct 30 00:19:08.685167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.684744 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9f81abc5-4bea-48b9-bd03-02012f98883a" containerName="main" Oct 30 00:19:08.685167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.684766 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f81abc5-4bea-48b9-bd03-02012f98883a" containerName="main" Oct 30 00:19:08.685167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.684925 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f81abc5-4bea-48b9-bd03-02012f98883a" containerName="main" Oct 30 00:19:08.688948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.688922 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.710610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.709380 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z"] Oct 30 00:19:08.729421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729389 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbl82\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-kube-api-access-gbl82\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.729526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729436 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.729589 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729549 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.729649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729599 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.729709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729653 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.729709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729701 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6ca814-0703-421b-afdd-dfe9b05555ea-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.729903 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729887 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.729963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729940 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.730015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.729995 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.775622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.775042 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7a7ac6-openshift-default-dockercfg-rv5lp\"" Oct 30 00:19:08.831029 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.830998 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831044 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831064 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831158 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbl82\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-kube-api-access-gbl82\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831182 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831234 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831263 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831288 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831325 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6ca814-0703-421b-afdd-dfe9b05555ea-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831506 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831634 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831775 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.831864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.831817 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.832090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.832065 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6ca814-0703-421b-afdd-dfe9b05555ea-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.833544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.833520 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.833717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.833699 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.880596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.880360 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbl82\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-kube-api-access-gbl82\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:08.881998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:08.881952 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:09.007288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.007197 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:09.016942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.016915 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:19:09.116410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.116376 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:09.407864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.407833 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z"] Oct 30 00:19:09.409064 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:19:09.409036 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6ca814_0703_421b_afdd_dfe9b05555ea.slice/crio-1c67a7808615bd3c50824394b7bb3f4df0e87f9ade753dd5704b1df4aece5e91 WatchSource:0}: Error finding container 1c67a7808615bd3c50824394b7bb3f4df0e87f9ade753dd5704b1df4aece5e91: Status 404 returned error can't find the container with id 1c67a7808615bd3c50824394b7bb3f4df0e87f9ade753dd5704b1df4aece5e91 Oct 30 00:19:09.411699 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.411663 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:09.411760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.411726 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:09.411815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.411757 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:09.496337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.496305 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:19:09.496484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:09.496360 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:19:10.023057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:10.023002 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" event={"ID":"4c6ca814-0703-421b-afdd-dfe9b05555ea","Type":"ContainerStarted","Data":"2e451e6cd99d714cd32731fc6e284d9832ba7dffdac0f51f90307ad00ded0fc9"} Oct 30 00:19:10.023632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:10.023075 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" event={"ID":"4c6ca814-0703-421b-afdd-dfe9b05555ea","Type":"ContainerStarted","Data":"1c67a7808615bd3c50824394b7bb3f4df0e87f9ade753dd5704b1df4aece5e91"} Oct 30 00:19:10.105939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:10.105725 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podStartSLOduration=2.105702216 podStartE2EDuration="2.105702216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:10.094750627 +0000 UTC m=+859.028819163" watchObservedRunningTime="2026-10-30 00:19:10.105702216 +0000 UTC m=+859.039770752" Oct 30 00:19:10.503664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:10.502050 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:19:10.503664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:10.502114 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:19:10.871597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:10.871526 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:11.008892 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.008861 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:11.010410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.010378 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:19:11.010551 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.010438 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:19:11.495938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.495905 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:19:11.496429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.495952 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:19:11.699515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.699480 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:11.819527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.819497 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:11.926765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:11.926736 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:12.007849 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:12.007805 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:19:12.008040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:12.007884 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:19:12.501870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:12.501833 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:12.502326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:12.502208 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:12.503174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:12.503153 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:19:12.714182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:12.714143 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:12.987047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:12.987015 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:13.008353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:13.008320 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:19:13.008516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:13.008380 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:19:13.098603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:13.098538 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:13.223123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:13.223088 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:14.007933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:14.007897 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:19:14.008450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:14.007960 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:19:14.477605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:14.477539 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:15.007522 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:15.007483 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:19:15.007734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:15.007549 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:19:16.007874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.007836 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:19:16.008271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.007900 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:19:16.139230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.139194 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:16.472477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.472442 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m"] Oct 30 00:19:16.477854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.477831 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.510333 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.509488 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m"] Oct 30 00:19:16.609723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.609690 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.609874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.609729 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.609874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.609833 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.609874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.609856 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.609987 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.609897 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05714c1e-3593-4b81-a683-fcba9d3438a7-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.610119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.610094 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.610157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.610131 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.610244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.610221 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.610296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.610261 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6qw\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-kube-api-access-pq6qw\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711301 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711270 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05714c1e-3593-4b81-a683-fcba9d3438a7-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711375 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711392 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711423 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711448 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6qw\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-kube-api-access-pq6qw\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711512 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711536 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711644 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711685 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.711905 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711846 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.712015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.711988 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.712087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.712071 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.712146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.712095 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.712146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.712106 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05714c1e-3593-4b81-a683-fcba9d3438a7-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.713876 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.713850 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.714029 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.714009 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.734983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.734936 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.735072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.735006 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6qw\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-kube-api-access-pq6qw\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:16.812832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:16.812805 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:17.012006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.011930 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:17.012392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.012312 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:17.012998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.012977 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:19:17.135734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.135707 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m"] Oct 30 00:19:17.141561 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:19:17.140633 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05714c1e_3593_4b81_a683_fcba9d3438a7.slice/crio-18ac5322ee15e97126d76a1c95b8a410d06a9aed7dd738c52fe0e54f91037e7e WatchSource:0}: Error finding container 18ac5322ee15e97126d76a1c95b8a410d06a9aed7dd738c52fe0e54f91037e7e: Status 404 returned error can't find the container with id 18ac5322ee15e97126d76a1c95b8a410d06a9aed7dd738c52fe0e54f91037e7e Oct 30 00:19:17.142907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.142835 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:17.143012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.142931 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:17.143012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.142977 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:17.691777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:17.691740 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:18.102501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:18.102464 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" event={"ID":"05714c1e-3593-4b81-a683-fcba9d3438a7","Type":"ContainerStarted","Data":"806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351"} Oct 30 00:19:18.102946 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:18.102529 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" event={"ID":"05714c1e-3593-4b81-a683-fcba9d3438a7","Type":"ContainerStarted","Data":"18ac5322ee15e97126d76a1c95b8a410d06a9aed7dd738c52fe0e54f91037e7e"} Oct 30 00:19:18.184686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:18.184627 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podStartSLOduration=2.184613039 podStartE2EDuration="2.184613039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:18.171252528 +0000 UTC m=+867.105321077" watchObservedRunningTime="2026-10-30 00:19:18.184613039 +0000 UTC m=+867.118681574" Oct 30 00:19:18.813804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:18.813765 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:18.815627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:18.815600 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:18.815750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:18.815648 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:19.605600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:19.589769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:19.814109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:19.814073 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:19.814386 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:19.814360 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:20.829598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:20.829200 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:20.829598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:20.829279 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:21.046683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.046216 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.176653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.174272 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:19:21.429780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.429678 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.501594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.496844 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.635064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.635020 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.635621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.635591 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:19:21.636036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.636008 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:19:21.746653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.745879 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.798598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.795068 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.837830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.835746 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.837830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.836326 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:21.837830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.836365 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:21.976598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:21.951671 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.097081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.097040 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:19:22.097961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.097936 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:19:22.153635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.130688 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.177596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.177501 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.182595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.178559 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:19:22.370863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.370753 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:19:22.371199 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.371079 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.475599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.463867 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.475599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.465766 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:19:22.502597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.497162 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.502597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.497496 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:19:22.534381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.534347 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:22.733833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.733738 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.762521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.719205 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.762521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.744869 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:19:22.762521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.744950 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:19:22.762521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.736719 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.762521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.760010 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:19:22.762521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.761679 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:19:22.762521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.761928 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:19:22.862597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.861711 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:19:22.863098 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.862669 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:19:22.865382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.865347 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.871343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.871313 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:22.871654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.871625 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:22.905613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.879659 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:19:22.963323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.960274 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.976032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:22.975997 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:19:23.026042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.014699 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 30 00:19:23.145609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.145551 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.149817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.149781 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.194702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.194657 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:19:23.436624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.430668 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:19:23.436624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.430733 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.549591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.467802 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.575597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.541333 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:19:23.858184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.858136 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:19:23.862604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.859906 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:19:23.862604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.860380 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:19:23.862604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.862445 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.873695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.873664 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.879992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.879967 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:24.050991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.050945 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:24.106271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.028006 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:19:24.113975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:23.937386 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.150637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.150227 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.207599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.197781 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.315136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.314035 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:19:24.315136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.314141 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:19:24.329082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.329056 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:19:24.330761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.330740 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:19:24.333858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.333837 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:19:24.340155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.340125 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:19:24.342598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.342518 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:24.347206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.347189 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:19:24.389838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.389762 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:19:24.389838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.389828 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:19:24.392807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.392784 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:19:24.813230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.813200 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:24.813413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:24.813254 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:25.813351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:25.813317 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:19:25.813771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:25.813379 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:19:25.896503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:25.896434 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:27.274117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:27.274086 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:27.275860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:27.275834 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:27.277354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:27.277321 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:19:27.555360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:27.555264 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:29.089937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:29.089906 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:30.675101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:30.675061 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:32.273660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:32.273614 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:32.665282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:32.665250 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:33.796707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:33.796677 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:35.162319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:35.162266 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:35.291791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:35.291748 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:36.842714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:36.842677 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:36.948119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:36.948085 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:37.102368 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:19:37.102275 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:19:38.475169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:38.475135 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:39.987673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:39.987625 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:41.467956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:41.467918 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:41.580466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:41.580435 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:43.077544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:43.077513 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:44.249568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.249532 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg"] Oct 30 00:19:44.254201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.254168 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.283044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.283012 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg"] Oct 30 00:19:44.378595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.378540 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.378799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.378607 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.378799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.378691 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bnm\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-kube-api-access-g4bnm\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.378799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.378752 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.378799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.378789 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.379050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.378905 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.379050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.379007 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.379050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.379049 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.379221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.379084 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.480617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480588 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.480617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480620 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.480867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480644 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bnm\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-kube-api-access-g4bnm\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.480867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480682 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.480867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480730 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.480867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480818 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480917 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.480976 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.481018 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.481025 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.481128 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.481233 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.481423 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.481682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.481662 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.483011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.482991 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.483493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.483473 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.504265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.504210 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.504349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.504317 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bnm\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-kube-api-access-g4bnm\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.592109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.592076 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:44.675660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.674533 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:44.918313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.918287 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg"] Oct 30 00:19:44.919420 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:19:44.919391 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b54a68c_7e52_4ed1_8c16_f46783a8a4cf.slice/crio-9d163f85cec8200daed6cef462980ae1c69308ca7a491acea79e62c662b7da4a WatchSource:0}: Error finding container 9d163f85cec8200daed6cef462980ae1c69308ca7a491acea79e62c662b7da4a: Status 404 returned error can't find the container with id 9d163f85cec8200daed6cef462980ae1c69308ca7a491acea79e62c662b7da4a Oct 30 00:19:44.921975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.921941 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:44.922042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.922013 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:44.922082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:44.922043 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:45.299902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:45.299810 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" event={"ID":"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf","Type":"ContainerStarted","Data":"dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f"} Oct 30 00:19:45.299902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:45.299876 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" event={"ID":"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf","Type":"ContainerStarted","Data":"9d163f85cec8200daed6cef462980ae1c69308ca7a491acea79e62c662b7da4a"} Oct 30 00:19:45.362662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:45.362615 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podStartSLOduration=1.362601747 podStartE2EDuration="1.362601747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:45.359836544 +0000 UTC m=+894.293905079" watchObservedRunningTime="2026-10-30 00:19:45.362601747 +0000 UTC m=+894.296670273" Oct 30 00:19:45.595495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:45.595456 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:45.599447 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:45.599402 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:45.599667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:45.599486 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:46.304627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:46.304588 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:46.592980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:46.592941 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:46.593165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:46.592996 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:47.595598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:47.595343 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:47.595598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:47.595413 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:48.192594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:48.192478 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:48.612597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:48.603828 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:48.613198 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:48.613161 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:49.152730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.152683 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.365654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.353129 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.365654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.353653 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.365654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.353735 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 00:19:49.368906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.366109 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.379616 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.373682 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 00:19:49.419977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.419188 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.489598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.443183 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t"] Oct 30 00:19:49.644341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.574412 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.737626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.658637 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:19:49.737626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.680701 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t"] Oct 30 00:19:49.737626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.680855 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.756783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.744975 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:49.756783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.745044 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:49.807096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.807062 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.810695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.810645 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.811457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.811435 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3493edb6-a29a-43ce-a229-5dde741d4733-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.826594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.817934 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgjz\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-kube-api-access-fmgjz\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.826594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.818095 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.826594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.818237 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.826594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.818375 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.826594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.818567 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3493edb6-a29a-43ce-a229-5dde741d4733-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.826594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.818729 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.868294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.865671 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.868294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.866318 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980476 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980532 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980642 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3493edb6-a29a-43ce-a229-5dde741d4733-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980684 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgjz\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-kube-api-access-fmgjz\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980731 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980775 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980818 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980903 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3493edb6-a29a-43ce-a229-5dde741d4733-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:49.982598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:49.980962 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.038650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.006027 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.038650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.006338 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.038650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.006933 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.038650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.007625 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3493edb6-a29a-43ce-a229-5dde741d4733-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.038650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.008202 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.038650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.034952 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 30 00:19:50.039251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.039183 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.142439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.088668 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.142439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.133687 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.195492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.168671 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:19:50.195492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.169706 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:19:50.278250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.210318 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3493edb6-a29a-43ce-a229-5dde741d4733-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.278250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.224443 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgjz\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-kube-api-access-fmgjz\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.278250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.225707 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.278250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.243115 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-d469t\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.386218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.353336 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:19:50.386218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.353592 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:19:50.386218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.355112 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.386218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.364525 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:19:50.416093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.416047 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:19:50.444275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.444243 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:19:50.445479 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.435777 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:19:50.445889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.435715 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:50.446812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.446003 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:19:50.639702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.637698 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.639702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.639239 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:19:50.665774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.665568 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.667313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.667293 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:19:50.690749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.673979 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:50.690749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.674023 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:50.852136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.741982 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.874894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.874853 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:51.020067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.878649 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:51.020067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.952846 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:19:51.020067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:50.952906 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:19:51.172609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.143773 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:51.172609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.144564 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:19:51.176600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.174710 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:19:51.254599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.249703 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:51.439655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.415917 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:19:51.455598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.449657 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:51.457178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.457138 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:51.488640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.460651 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:51.736097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.735974 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.068604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.981679 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.068604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:51.997878 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:19:52.087600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.080043 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.190125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.088135 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:52.190420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.190381 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:52.190590 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.129086 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.257943 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.257902 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.293768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.293659 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.361659 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:19:52.353516 4433 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.068s" Oct 30 00:19:52.361659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.356959 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:19:52.467607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.452862 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:19:52.596604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.542638 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:19:52.635659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.630442 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:52.635659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.630543 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:52.713800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.713563 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.757991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.757944 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:19:52.810599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:52.795847 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 00:19:53.024832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.022793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:53.088148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.088096 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 30 00:19:53.402595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.376612 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 00:19:53.600807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.600705 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:19:53.627733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.627690 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:19:53.647326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.647285 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:19:53.663598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.649525 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.665735 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.665697 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.688645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.671204 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:19:53.688645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.673667 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:19:53.715751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.715716 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:53.716835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.716797 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.762852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:53.762775 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:19:54.038668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.038546 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:19:54.055223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.055184 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:54.063705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.063663 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:54.128249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.128211 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t"] Oct 30 00:19:54.135304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.135264 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 00:19:54.249395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.249355 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:54.274063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.274015 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:54.274264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.274114 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:54.274264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.274162 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:19:54.614676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.614596 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:19:54.614676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.614662 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" event={"ID":"3493edb6-a29a-43ce-a229-5dde741d4733","Type":"ContainerStarted","Data":"1ccb61e7a32500a691a62b832e4427552be5fbebcc1c221f3a14722d3a07f4d0"} Oct 30 00:19:54.667363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.615661 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:19:54.667363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.622594 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:54.667363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.622644 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:54.667363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.632064 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.667363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.666678 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.667363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.667068 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:19:54.765761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.765658 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.773913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.766997 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.774305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.767527 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.777866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.767587 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.807497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.807463 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:54.837640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:54.836115 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:19:55.048745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.039870 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:19:55.086591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.077831 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:19:55.086591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.080694 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:55.110645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.108829 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:19:55.135996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.135949 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 00:19:55.145326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.145289 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:19:55.171254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.170822 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:19:55.219668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.217592 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:19:55.219668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.217645 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:19:55.265371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.265160 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:19:55.271234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.270279 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:19:55.294174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.294149 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:19:55.305344 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.305274 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:19:55.386825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.386791 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:55.557485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.557454 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" event={"ID":"3493edb6-a29a-43ce-a229-5dde741d4733","Type":"ContainerStarted","Data":"85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5"} Oct 30 00:19:55.593014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.592978 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:19:55.593174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.593041 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:19:55.604239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:55.604186 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podStartSLOduration=6.604173572 podStartE2EDuration="6.604173572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:55.603804998 +0000 UTC m=+904.537873554" watchObservedRunningTime="2026-10-30 00:19:55.604173572 +0000 UTC m=+904.538242106" Oct 30 00:19:56.437598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:56.436732 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:19:56.440620 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:56.439278 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-d469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:19:56.440620 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:56.439329 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 00:19:57.094851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.093325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:57.437042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.436959 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-d469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:19:57.437042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.437008 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 00:19:57.633733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.633696 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:19:57.634133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.633757 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:57.734112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.734030 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:57.734359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.734337 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:57.735677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:57.735655 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:19:58.436766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.436730 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-d469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:19:58.437027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.436794 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 00:19:58.496891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.496776 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8"] Oct 30 00:19:58.502964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.502745 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8"] Oct 30 00:19:58.502964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.502854 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.521614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.521428 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"default-dockercfg-s9svd\"" Oct 30 00:19:58.521614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.521515 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"openshift-service-ca.crt\"" Oct 30 00:19:58.527461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.527424 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:58.534338 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.534317 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"kube-root-ca.crt\"" Oct 30 00:19:58.534615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.534560 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"distinct-2-w0-7505bd-kserve-self-signed-certs\"" Oct 30 00:19:58.559704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.551839 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rv4r\" (UniqueName: \"kubernetes.io/projected/2e586cc2-705e-44fb-abaa-dd8993da8faf-kube-api-access-5rv4r\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.559704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.552069 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-model-cache\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.559704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.552109 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e586cc2-705e-44fb-abaa-dd8993da8faf-tls-certs\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.559704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.552203 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-dshm\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.559704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.552290 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-home\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.559704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.552319 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-tmp-dir\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.666638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.665695 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-model-cache\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.666638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.665760 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e586cc2-705e-44fb-abaa-dd8993da8faf-tls-certs\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.666638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.665857 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-dshm\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.666638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.665941 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-home\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.666638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.665968 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-tmp-dir\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.666638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.666022 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rv4r\" (UniqueName: \"kubernetes.io/projected/2e586cc2-705e-44fb-abaa-dd8993da8faf-kube-api-access-5rv4r\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.691388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.682722 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-home\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.691734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.683960 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-tmp-dir\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.691934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.689976 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-model-cache\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.700235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.697391 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e586cc2-705e-44fb-abaa-dd8993da8faf-tls-certs\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.700235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.700070 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-dshm\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.711978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.711942 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rv4r\" (UniqueName: \"kubernetes.io/projected/2e586cc2-705e-44fb-abaa-dd8993da8faf-kube-api-access-5rv4r\") pod \"distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:58.728234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.728199 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:19:58.820690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:58.820647 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:19:59.181599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:59.170483 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8"] Oct 30 00:19:59.436161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:59.436069 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-d469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:19:59.436161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:59.436129 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 00:19:59.619264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:59.619215 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" event={"ID":"2e586cc2-705e-44fb-abaa-dd8993da8faf","Type":"ContainerStarted","Data":"af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651"} Oct 30 00:19:59.619264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:59.619269 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" event={"ID":"2e586cc2-705e-44fb-abaa-dd8993da8faf","Type":"ContainerStarted","Data":"9b51fff05f3f04432dd05c123e9d2dd0b9a51148ab9b9341723e395f3ed1e138"} Oct 30 00:19:59.821246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:19:59.821214 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:20:00.424274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:00.424239 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:00.436879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:00.436850 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-d469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:20:00.437024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:00.436897 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 00:20:01.436440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:01.436405 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-d469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:20:01.436943 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:01.436466 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 00:20:01.498609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:01.498552 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:01.602738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:01.602694 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:02.029249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:02.029205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:02.290469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:02.290381 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:02.436724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:02.436682 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-d469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:20:02.437173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:02.436765 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 00:20:03.148552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:03.148512 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:03.255497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:03.255459 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:03.519417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:03.519348 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:20:03.520677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:03.520523 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:20:03.521287 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:03.521265 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:20:03.581552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:03.581499 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podStartSLOduration=5.581481697 podStartE2EDuration="5.581481697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:59.676851375 +0000 UTC m=+908.610919935" watchObservedRunningTime="2026-10-30 00:20:03.581481697 +0000 UTC m=+912.515550294" Oct 30 00:20:03.972222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:03.972191 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:05.565031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:05.564993 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:05.992227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:05.992134 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2"] Oct 30 00:20:05.996564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:05.996544 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.026513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.026476 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2"] Oct 30 00:20:06.112921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.112880 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.112935 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.113206 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113346 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.113326 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.113357 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6pc\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-kube-api-access-gm6pc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.113451 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.113553 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.113665 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.113746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.113702 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.215518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.215482 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.215713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.215583 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.215713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.215617 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.215713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.215667 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.215725 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.215754 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.215927 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.216018 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.216051 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6pc\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-kube-api-access-gm6pc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.216102 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.216286 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.216466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.216399 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.217336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.217299 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.217477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.217438 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.219764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.219736 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.220891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.220860 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.240439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.240412 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.240787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.240767 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6pc\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-kube-api-access-gm6pc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ts2j2\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.312469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.312440 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:06.643308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.643281 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2"] Oct 30 00:20:06.645463 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:20:06.645432 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d148256_ff4d_48eb_b0a6_1c28033d6b51.slice/crio-4c8fe205c8627c304af2c61dcec1391186aba79322c9b4ddd8aa10f2f91cf184 WatchSource:0}: Error finding container 4c8fe205c8627c304af2c61dcec1391186aba79322c9b4ddd8aa10f2f91cf184: Status 404 returned error can't find the container with id 4c8fe205c8627c304af2c61dcec1391186aba79322c9b4ddd8aa10f2f91cf184 Oct 30 00:20:06.648023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.647991 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:06.648104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.648055 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:06.648104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.648082 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:06.681159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:06.681124 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" event={"ID":"9d148256-ff4d-48eb-b0a6-1c28033d6b51","Type":"ContainerStarted","Data":"4c8fe205c8627c304af2c61dcec1391186aba79322c9b4ddd8aa10f2f91cf184"} Oct 30 00:20:07.091234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:07.091196 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:07.102525 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:20:07.102488 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:20:07.689155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:07.689115 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" event={"ID":"9d148256-ff4d-48eb-b0a6-1c28033d6b51","Type":"ContainerStarted","Data":"2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f"} Oct 30 00:20:07.750827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:07.750774 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podStartSLOduration=2.7507587989999998 podStartE2EDuration="2.750758799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:07.748306428 +0000 UTC m=+916.682374986" watchObservedRunningTime="2026-10-30 00:20:07.750758799 +0000 UTC m=+916.684827391" Oct 30 00:20:08.312646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.312608 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:08.313824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.313799 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:08.313954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.313849 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:08.702300 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.702208 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:08.821715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.821681 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:20:08.835766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.835734 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:20:08.983801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.983689 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf"] Oct 30 00:20:08.988867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:08.988834 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.009560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.009520 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf"] Oct 30 00:20:09.083335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083292 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083346 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083475 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083524 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083566 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083680 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083716 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083747 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjb5m\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-kube-api-access-mjb5m\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.083938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.083779 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185142 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185196 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185238 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185273 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185328 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185427 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185462 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185491 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjb5m\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-kube-api-access-mjb5m\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185522 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185641 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.185823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.185655 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.186112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.186082 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.186112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.186101 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.186315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.186285 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.187911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.187887 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.188178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.188161 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.209534 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.209500 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.209765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.209740 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjb5m\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-kube-api-access-mjb5m\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.313800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.313764 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:09.313963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.313827 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:09.318070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.318040 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:09.382068 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.381975 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:09.680473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.680440 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf"] Oct 30 00:20:09.683209 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:20:09.683171 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191e9810_53a6_41a4_b6fd_aa2f8b9a5faa.slice/crio-72ac1f5c238102320a5a9be63d42cff1bbe8fa7d7c9a1c036e64e25d4a2afd1e WatchSource:0}: Error finding container 72ac1f5c238102320a5a9be63d42cff1bbe8fa7d7c9a1c036e64e25d4a2afd1e: Status 404 returned error can't find the container with id 72ac1f5c238102320a5a9be63d42cff1bbe8fa7d7c9a1c036e64e25d4a2afd1e Oct 30 00:20:09.686048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.686009 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:09.686155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.686098 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:09.686155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.686148 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:09.706435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.706403 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" event={"ID":"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa","Type":"ContainerStarted","Data":"72ac1f5c238102320a5a9be63d42cff1bbe8fa7d7c9a1c036e64e25d4a2afd1e"} Oct 30 00:20:09.719550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.719527 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:20:09.807378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:09.807329 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:10.313465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.313427 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:10.313662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.313489 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:10.483768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.483720 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:10.529227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.529195 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47"] Oct 30 00:20:10.533816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.533790 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.549880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.548605 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47"] Oct 30 00:20:10.591679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.590829 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:10.641122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641086 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641253 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641301 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91ac229-73ad-4732-b919-9d35d165227c-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641487 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641468 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641508 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91ac229-73ad-4732-b919-9d35d165227c-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641616 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641600 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8wcv\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-kube-api-access-k8wcv\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641676 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641797 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.641879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.641838 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.722887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.722845 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" event={"ID":"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa","Type":"ContainerStarted","Data":"de1d8d6e7f4b574994a9709ad1ede354d68a6b17d23290fc42860a3bf807daf2"} Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.746519 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.746615 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91ac229-73ad-4732-b919-9d35d165227c-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.746691 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8wcv\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-kube-api-access-k8wcv\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.746857 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.747010 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.747051 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.747131 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.747273 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.747327 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91ac229-73ad-4732-b919-9d35d165227c-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.749518 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.749797 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.750597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.750032 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.751441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.751099 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.769601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.751787 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91ac229-73ad-4732-b919-9d35d165227c-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.773599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.772884 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.807748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.804823 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podStartSLOduration=2.804800985 podStartE2EDuration="2.804800985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:10.797007005 +0000 UTC m=+919.731075557" watchObservedRunningTime="2026-10-30 00:20:10.804800985 +0000 UTC m=+919.738869520" Oct 30 00:20:10.818712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.814642 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91ac229-73ad-4732-b919-9d35d165227c-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.831646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.830151 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.832123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.832094 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8wcv\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-kube-api-access-k8wcv\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:10.891652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:10.890940 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:11.311497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.311464 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47"] Oct 30 00:20:11.313454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.313431 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:11.313673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.313653 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:11.315157 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:20:11.315132 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91ac229_73ad_4732_b919_9d35d165227c.slice/crio-26415e3e27b653aae7579644a0b83a066b6db33e00d8c2595128c26a2c3711b0 WatchSource:0}: Error finding container 26415e3e27b653aae7579644a0b83a066b6db33e00d8c2595128c26a2c3711b0: Status 404 returned error can't find the container with id 26415e3e27b653aae7579644a0b83a066b6db33e00d8c2595128c26a2c3711b0 Oct 30 00:20:11.318074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.318044 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:11.318140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.318107 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:11.318174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.318138 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:11.318524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.318511 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:11.319738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.319667 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:20:11.319738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.319703 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:20:11.731342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.731257 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" event={"ID":"a91ac229-73ad-4732-b919-9d35d165227c","Type":"ContainerStarted","Data":"436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382"} Oct 30 00:20:11.731342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.731304 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" event={"ID":"a91ac229-73ad-4732-b919-9d35d165227c","Type":"ContainerStarted","Data":"26415e3e27b653aae7579644a0b83a066b6db33e00d8c2595128c26a2c3711b0"} Oct 30 00:20:11.827390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.827299 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podStartSLOduration=1.827265153 podStartE2EDuration="1.827265153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:11.777166113 +0000 UTC m=+920.711234649" watchObservedRunningTime="2026-10-30 00:20:11.827265153 +0000 UTC m=+920.761333688" Oct 30 00:20:11.913348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:11.902750 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:12.313598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:12.312956 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:12.313598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:12.313021 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:12.324595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:12.319270 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:20:12.324595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:12.319326 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:20:12.434036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:12.433725 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:12.913598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:12.913286 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:20:12.913598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:12.913366 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 00:20:13.221010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.220762 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:13.314402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.314367 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:13.314709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.314681 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:13.320592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.320490 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:20:13.320715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.320610 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:20:13.329071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.329041 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:13.898600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.896977 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:20:13.898600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:13.897046 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 00:20:14.321289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.317085 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:14.321289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.317145 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:14.321289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.317906 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4"] Oct 30 00:20:14.321289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.321152 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:20:14.321289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.321193 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:20:14.323369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.323349 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.338684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.338655 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4"] Oct 30 00:20:14.357601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.347642 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3fae6d-serving-cert\"" Oct 30 00:20:14.427627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.424012 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:14.444562 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.444498 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-metrics-certs\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.444782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.444562 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzzk\" (UniqueName: \"kubernetes.io/projected/b619fe16-234d-4a85-b829-39325260091d-kube-api-access-rzzzk\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.444782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.444729 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-maas-api-tls\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.444927 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.444855 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b619fe16-234d-4a85-b829-39325260091d-openshift-service-ca\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.550814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.550709 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:14.631682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.552322 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b619fe16-234d-4a85-b829-39325260091d-openshift-service-ca\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.654606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.647351 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-metrics-certs\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.654606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.647407 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzzk\" (UniqueName: \"kubernetes.io/projected/b619fe16-234d-4a85-b829-39325260091d-kube-api-access-rzzzk\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.654606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.647612 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-maas-api-tls\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.654606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.556417 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b619fe16-234d-4a85-b829-39325260091d-openshift-service-ca\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.696669 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.687811 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-maas-api-tls\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.707673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.704628 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:14.725969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.725934 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-metrics-certs\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.741494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.741169 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzzk\" (UniqueName: \"kubernetes.io/projected/b619fe16-234d-4a85-b829-39325260091d-kube-api-access-rzzzk\") pod \"maas-api-e2e-api-a-3fae6d-8856c7455-v64f4\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:14.894654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.894013 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:20:14.894654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.894077 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 00:20:14.938972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:14.938930 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:15.342628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.332843 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:20:15.342628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.332905 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:20:15.342628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.338626 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:20:15.342628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.338681 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 00:20:15.427892 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.427856 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4"] Oct 30 00:20:15.431763 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:20:15.431730 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb619fe16_234d_4a85_b829_39325260091d.slice/crio-a237abeb6a34799d29d13a1e3d777037898b71c7af7bc570def30b02fcb15d83 WatchSource:0}: Error finding container a237abeb6a34799d29d13a1e3d777037898b71c7af7bc570def30b02fcb15d83: Status 404 returned error can't find the container with id a237abeb6a34799d29d13a1e3d777037898b71c7af7bc570def30b02fcb15d83 Oct 30 00:20:15.763033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.762888 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:15.906246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.906191 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:15.947660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.947623 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:20:15.947954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:15.947903 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 00:20:16.103634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.103476 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" event={"ID":"b619fe16-234d-4a85-b829-39325260091d","Type":"ContainerStarted","Data":"a237abeb6a34799d29d13a1e3d777037898b71c7af7bc570def30b02fcb15d83"} Oct 30 00:20:16.361518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.357013 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:20:16.361518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.357086 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:20:16.390769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.390217 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb"] Oct 30 00:20:16.475785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.452811 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.535775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.534385 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb"] Oct 30 00:20:16.544963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.544941 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 00:20:16.545163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.545151 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 00:20:16.545427 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.545413 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nht5w\"" Oct 30 00:20:16.545567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.545556 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 00:20:16.587597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.584851 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.587597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.584991 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.598551 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.598512 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f5c4a-f002-4e25-8c88-51082bf3417d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.600066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.600038 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.617417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.600268 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.617417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.613826 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.617417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.613866 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wg4k\" (UniqueName: \"kubernetes.io/projected/db7f5c4a-f002-4e25-8c88-51082bf3417d-kube-api-access-2wg4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.641812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.641779 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n"] Oct 30 00:20:16.664633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.662864 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.709634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.708550 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n"] Oct 30 00:20:16.715718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.715277 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.716245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.716211 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wg4k\" (UniqueName: \"kubernetes.io/projected/db7f5c4a-f002-4e25-8c88-51082bf3417d-kube-api-access-2wg4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.716366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.716310 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.716448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.716428 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.716715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.716693 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f5c4a-f002-4e25-8c88-51082bf3417d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.716879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.716863 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.717002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.716987 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.718149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.718126 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.718613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.718592 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:16.719516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.719418 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.722282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.722234 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.722893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.722871 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.724934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.724887 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f5c4a-f002-4e25-8c88-51082bf3417d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.780604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.772322 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f5c4a-f002-4e25-8c88-51082bf3417d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.819650 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.819700 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4s5\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-kube-api-access-dw4s5\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.819735 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.819840 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.819915 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.819961 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.819986 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3993addf-2a45-44a7-9714-ad89b90ea081-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.820181 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3993addf-2a45-44a7-9714-ad89b90ea081-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.828601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.820277 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.860598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.851543 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wg4k\" (UniqueName: \"kubernetes.io/projected/db7f5c4a-f002-4e25-8c88-51082bf3417d-kube-api-access-2wg4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vtwgb\" (UID: \"db7f5c4a-f002-4e25-8c88-51082bf3417d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:16.895663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.894027 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:20:16.895663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.894091 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 00:20:16.921398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921264 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921303 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4s5\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-kube-api-access-dw4s5\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921322 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921382 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921424 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921451 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921467 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3993addf-2a45-44a7-9714-ad89b90ea081-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921582 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3993addf-2a45-44a7-9714-ad89b90ea081-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.921734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.921652 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.922453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.922307 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.922453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.922392 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.922634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.922490 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.922965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.922922 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.923513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.923458 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3993addf-2a45-44a7-9714-ad89b90ea081-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.924755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.924709 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.925345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.925304 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3993addf-2a45-44a7-9714-ad89b90ea081-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.946420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.946378 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:16.946944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:16.946918 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4s5\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-kube-api-access-dw4s5\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:17.116871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.116836 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:17.119006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.118964 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" event={"ID":"b619fe16-234d-4a85-b829-39325260091d","Type":"ContainerStarted","Data":"9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5"} Oct 30 00:20:17.119195 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.119176 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:17.135440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.135418 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:17.220181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.220073 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" podStartSLOduration=2.231728591 podStartE2EDuration="3.220051369s" totalImagesPullingTime="988.322778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:17.206090611 +0000 UTC m=+926.140159146" watchObservedRunningTime="2026-10-30 00:20:17.220051369 +0000 UTC m=+926.154119904" Oct 30 00:20:17.318739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.318492 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:20:17.318739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.318557 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:20:17.415265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.400017 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:20:17.415265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.400088 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:17.494868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.494823 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:17.503708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.496682 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:17.506609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.506542 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:20:17.544304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.544255 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n"] Oct 30 00:20:17.548652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.548605 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:17.548774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.548697 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:17.548774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.548745 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:17.624850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.624826 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb"] Oct 30 00:20:17.630092 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:20:17.630063 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7f5c4a_f002_4e25_8c88_51082bf3417d.slice/crio-daf1a722763bca49bef1b11fa4fe13f03d08c68bd47b9708d9ed6864f57e87f5 WatchSource:0}: Error finding container daf1a722763bca49bef1b11fa4fe13f03d08c68bd47b9708d9ed6864f57e87f5: Status 404 returned error can't find the container with id daf1a722763bca49bef1b11fa4fe13f03d08c68bd47b9708d9ed6864f57e87f5 Oct 30 00:20:17.635670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.635599 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:17.635929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.635900 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:20:17.896682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.896642 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:20:17.897004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:17.896963 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 00:20:18.145443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.145402 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" event={"ID":"db7f5c4a-f002-4e25-8c88-51082bf3417d","Type":"ContainerStarted","Data":"7c47c18a837376629161677b9c0a9a8169e112516e69b89b51b84f659547b438"} Oct 30 00:20:18.145715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.145687 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" event={"ID":"db7f5c4a-f002-4e25-8c88-51082bf3417d","Type":"ContainerStarted","Data":"daf1a722763bca49bef1b11fa4fe13f03d08c68bd47b9708d9ed6864f57e87f5"} Oct 30 00:20:18.148413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.148390 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:18.149481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.149459 4433 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vtwgb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:20:18.150229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.150206 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" podUID="db7f5c4a-f002-4e25-8c88-51082bf3417d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:20:18.152463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.152438 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" event={"ID":"3993addf-2a45-44a7-9714-ad89b90ea081","Type":"ContainerStarted","Data":"ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4"} Oct 30 00:20:18.153379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.153309 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" event={"ID":"3993addf-2a45-44a7-9714-ad89b90ea081","Type":"ContainerStarted","Data":"c3bbb16915811ce488180c833cade5f4eae68f5fa540651baf7bb16c96718bb4"} Oct 30 00:20:18.217640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.217531 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" podStartSLOduration=2.217505904 podStartE2EDuration="2.217505904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:18.207719543 +0000 UTC m=+927.141788109" watchObservedRunningTime="2026-10-30 00:20:18.217505904 +0000 UTC m=+927.151574442" Oct 30 00:20:18.255865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.255749 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podStartSLOduration=2.255729111 podStartE2EDuration="2.255729111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:18.253964237 +0000 UTC m=+927.188032772" watchObservedRunningTime="2026-10-30 00:20:18.255729111 +0000 UTC m=+927.189797647" Oct 30 00:20:18.629083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.629036 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:18.633120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.633090 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:18.663902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.663869 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:20:18.837594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.831370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:18.907653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.906848 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:20:18.907653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:18.906920 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 00:20:19.117560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.117522 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:19.119288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.119261 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:20:19.119418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.119312 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 00:20:19.181717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.181604 4433 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vtwgb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:20:19.182009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.181978 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" podUID="db7f5c4a-f002-4e25-8c88-51082bf3417d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:20:19.900968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.900931 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:19.901454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.901435 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:19.903488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:19.903460 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:20:20.117880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:20.117846 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:20:20.118017 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:20.117901 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 00:20:20.188269 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:20.188173 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vtwgb" Oct 30 00:20:20.777613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:20.777541 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:21.117902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:21.117866 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:20:21.118319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:21.117923 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 00:20:22.118144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:22.118109 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:20:22.118647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:22.118168 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 00:20:22.451020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:22.450935 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:23.117860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:23.117824 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:20:23.118040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:23.117879 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 00:20:23.195398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:23.195363 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:20:24.075497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:24.075461 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:24.117900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:24.117858 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:20:24.118047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:24.117932 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 00:20:25.124191 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:25.124159 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:25.124650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:25.124623 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:25.125470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:25.125444 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:20:25.715231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:25.715169 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:27.621750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:27.621705 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:28.015656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.008677 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:28.069730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.069677 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:28.070297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.070264 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:20:28.070769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.070743 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:20:28.189869 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.189831 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:20:28.194671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.194635 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:20:28.339711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.339661 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:28.444697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.444649 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:28.456699 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.456662 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:28.522194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:28.522149 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:29.446418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:29.446384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:29.558156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:29.558111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:31.082185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:31.082153 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:32.678104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:32.678066 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:34.354269 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:34.354229 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:35.981756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:35.981718 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:36.936074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:36.936037 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:37.042263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:37.042215 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:37.101320 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:20:37.101285 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:20:37.460132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:37.460085 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:37.623877 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:37.623830 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:38.104527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:38.104493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:38.235235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:38.235042 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:39.080310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:39.080275 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:40.656431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:40.656394 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:41.801786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:41.801747 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:41.907408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:41.907370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:42.389325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:42.389286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:43.492642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:43.492566 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:43.611368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:43.611327 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:44.138386 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:44.138347 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:45.333989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:45.333950 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:45.686852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:45.686745 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:45.806801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:45.806744 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:45.982388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:45.982295 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:46.177404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:46.177362 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:46.632724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:46.632682 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:46.776934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:46.776902 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:47.300627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:47.300539 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:48.679567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:48.679532 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:49.048587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:49.048494 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:50.325514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:50.325482 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:50.432094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:50.432055 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:50.538268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:50.538233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:50.652002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:50.651917 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:50.758800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:50.758768 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:50.863214 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:50.863175 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:51.985487 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:51.985436 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:52.095095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:52.095057 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:52.525527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:52.525489 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:52.871021 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:52.870984 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:54.228036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:54.228003 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:55.143598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:55.135708 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:55.484002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:55.483885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:55.773768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:55.773664 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:55.840496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:55.840453 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.090716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.090655 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:20:56.200474 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.199180 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:20:56.200474 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.199271 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.284819 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.283671 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:20:56.314253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.313206 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:20:56.489904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.489815 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:56.514352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.498691 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.515059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.510990 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.710740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.710691 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.783035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.782351 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.784533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:56.784506 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:20:57.085718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.085677 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:20:57.141597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.106554 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.141597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.106937 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.141597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.107398 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:20:57.185774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.184477 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.336148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.336055 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.349811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.347094 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.385339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.385307 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:20:57.385663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.385636 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:20:57.577796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.577754 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:20:57.604831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.604733 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:20:57.629683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.623740 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.630218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.630198 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:20:57.671350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.671231 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:20:57.711655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.710552 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:20:57.740094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.739979 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:20:57.747711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:57.747688 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:20:58.429914 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:58.429879 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:20:58.538207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:20:58.538175 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:00.079632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:00.079553 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:02.408350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:02.408319 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:03.261066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:03.261032 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:04.771681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:04.771645 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:05.343427 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.343394 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh"] Oct 30 00:21:05.347055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.347026 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.385084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.385039 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh"] Oct 30 00:21:05.389132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.389106 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f"] Oct 30 00:21:05.393916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.393894 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.404493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.404459 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f"] Oct 30 00:21:05.440404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440375 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440417 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440644 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440619 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d44d38e-54b0-473f-8f81-64328091c5f7-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440673 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440767 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440753 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440797 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tslgw\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-kube-api-access-tslgw\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440847 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440820 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440936 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440911 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.440987 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.440971 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543220 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543287 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tslgw\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-kube-api-access-tslgw\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543319 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543347 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543387 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543416 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.543782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543512 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.543782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543545 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.543782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543613 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.543782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543702 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543798 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfr4\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-kube-api-access-zzfr4\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.543994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543814 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543838 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.543994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543920 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.543994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543963 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.544275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.543995 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.544275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.544019 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.544275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.544133 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d44d38e-54b0-473f-8f81-64328091c5f7-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.544275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.544162 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.544275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.544262 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.544488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.544309 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.544706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.544684 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.544856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.544839 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d44d38e-54b0-473f-8f81-64328091c5f7-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.546011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.545993 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.546495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.546480 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.572254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.572226 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tslgw\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-kube-api-access-tslgw\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.572518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.572496 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.646229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646132 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646223 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646257 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646344 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfr4\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-kube-api-access-zzfr4\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646374 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646434 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646465 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646548 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646661 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646792 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.646910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.646897 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.647133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.647101 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.647259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.647159 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.647450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.647431 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.648727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.648702 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.648885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.648866 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.661221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.661191 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:05.670641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.670615 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.670964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.670942 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfr4\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-kube-api-access-zzfr4\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.711188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.711152 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:05.989891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.989674 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh"] Oct 30 00:21:05.993465 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:05.993433 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d44d38e_54b0_473f_8f81_64328091c5f7.slice/crio-335e4f94aec20e33dd09bca86b9c34fb4ecb0da062710994323e6c8af84325ff WatchSource:0}: Error finding container 335e4f94aec20e33dd09bca86b9c34fb4ecb0da062710994323e6c8af84325ff: Status 404 returned error can't find the container with id 335e4f94aec20e33dd09bca86b9c34fb4ecb0da062710994323e6c8af84325ff Oct 30 00:21:05.996468 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.996428 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:05.996557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.996506 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:05.996557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:05.996540 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:06.049527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.049499 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f"] Oct 30 00:21:06.050103 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:06.050078 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532bb5f8_3ce8_46e7_8dd9_e8e5cf03771f.slice/crio-61d856ba1121f488e5465ae332bcae3bb68c6092a5f6871cf062a0cb2e9ca08c WatchSource:0}: Error finding container 61d856ba1121f488e5465ae332bcae3bb68c6092a5f6871cf062a0cb2e9ca08c: Status 404 returned error can't find the container with id 61d856ba1121f488e5465ae332bcae3bb68c6092a5f6871cf062a0cb2e9ca08c Oct 30 00:21:06.053482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.053448 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:06.053562 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.053528 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:06.053685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.053651 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:06.375194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.375160 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:06.696092 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.696004 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" event={"ID":"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f","Type":"ContainerStarted","Data":"702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef"} Oct 30 00:21:06.696092 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.696060 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" event={"ID":"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f","Type":"ContainerStarted","Data":"61d856ba1121f488e5465ae332bcae3bb68c6092a5f6871cf062a0cb2e9ca08c"} Oct 30 00:21:06.697933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.697907 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" event={"ID":"2d44d38e-54b0-473f-8f81-64328091c5f7","Type":"ContainerStarted","Data":"3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49"} Oct 30 00:21:06.698069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.697950 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" event={"ID":"2d44d38e-54b0-473f-8f81-64328091c5f7","Type":"ContainerStarted","Data":"335e4f94aec20e33dd09bca86b9c34fb4ecb0da062710994323e6c8af84325ff"} Oct 30 00:21:06.711282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.711253 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:06.743648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.743604 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podStartSLOduration=1.74359129 podStartE2EDuration="1.74359129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:06.741419618 +0000 UTC m=+975.675488155" watchObservedRunningTime="2026-10-30 00:21:06.74359129 +0000 UTC m=+975.677659815" Oct 30 00:21:06.795796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:06.795741 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podStartSLOduration=1.795729083 podStartE2EDuration="1.795729083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:06.792828669 +0000 UTC m=+975.726897204" watchObservedRunningTime="2026-10-30 00:21:06.795729083 +0000 UTC m=+975.729797618" Oct 30 00:21:07.138859 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:21:07.138808 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:21:07.668596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:07.668307 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:07.668596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:07.668502 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:21:07.668596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:07.668552 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:21:07.715911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:07.715815 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:21:07.715911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:07.715886 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:21:08.018670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:08.018525 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:08.128592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:08.128546 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:08.240091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:08.240056 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:08.708484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:08.708442 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:21:08.708706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:08.708522 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:21:08.719944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:08.717642 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:21:08.719944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:08.717705 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:21:09.145355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.145212 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx"] Oct 30 00:21:09.149746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.149718 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.250485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.250437 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735ced8b-6586-4e93-8d8a-3439fab15f89-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.250939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.250525 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.250939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.250631 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.250939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.250715 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.250939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.250858 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.251149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.250953 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7vq\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-kube-api-access-6t7vq\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.251149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.251004 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.251251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.251166 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.251498 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.251472 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.324609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.323777 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx"] Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.362745 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.362872 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7vq\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-kube-api-access-6t7vq\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.362907 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.362991 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.363047 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.363114 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735ced8b-6586-4e93-8d8a-3439fab15f89-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.363175 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.363255 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.363600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.363333 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.391597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.390669 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735ced8b-6586-4e93-8d8a-3439fab15f89-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.391597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.390972 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.391597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.391542 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.416643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.392067 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.416643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.392329 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.421871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.417552 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7vq\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-kube-api-access-6t7vq\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.421871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.421092 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.421871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.421723 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.424074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.424051 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.630309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.630276 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:09.712605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.663967 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:21:09.712605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.664042 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:21:09.712605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.712249 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:21:09.712605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:09.712363 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:21:10.013647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.012999 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:10.043464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.043404 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx"] Oct 30 00:21:10.097107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.096106 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:10.097107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.096212 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:10.097107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.096261 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:10.664666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.662988 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:21:10.664666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.663054 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:21:10.712734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.712677 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:21:10.712937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.712749 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:21:10.823277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.823235 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" event={"ID":"735ced8b-6586-4e93-8d8a-3439fab15f89","Type":"ContainerStarted","Data":"25ba1b93d3b26148825a53714bb7e9d0e2ba9938dc9928fc15053df8321d005f"} Oct 30 00:21:10.823462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.823306 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" event={"ID":"735ced8b-6586-4e93-8d8a-3439fab15f89","Type":"ContainerStarted","Data":"7fa2293c0d4e1ea5d9b9317eba12206bc892e3872dc9476a8daabcfd441d7fc4"} Oct 30 00:21:10.929706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:10.919451 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podStartSLOduration=1.919430594 podStartE2EDuration="1.919430594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:10.917625039 +0000 UTC m=+979.851693580" watchObservedRunningTime="2026-10-30 00:21:10.919430594 +0000 UTC m=+979.853499129" Oct 30 00:21:11.643601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.642592 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:11.644997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.644962 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:21:11.645263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.645219 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:21:11.677222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.677185 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:21:11.677811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.677777 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:21:11.711904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.711868 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:21:11.712158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.712123 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:21:11.959762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:11.950407 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:12.096596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.071519 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:12.343719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.337125 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:12.642435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.642344 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:21:12.642711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.642673 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:21:12.664005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.663962 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:21:12.664295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.664255 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:21:12.711912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.711881 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:21:12.712376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:12.712351 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:21:13.631740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:13.631156 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:21:13.631740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:13.631222 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:21:13.673598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:13.673347 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:21:13.673598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:13.673409 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:21:13.713120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:13.712701 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:21:13.713120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:13.712762 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:21:14.018943 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.018747 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:14.631336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.631301 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:21:14.631514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.631360 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:21:14.682635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.672824 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:14.682635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.674394 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:14.682875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.682837 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:21:14.683686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.683649 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:14.715871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.715850 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:14.716247 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.716234 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:14.716988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:14.716969 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:21:15.630767 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:15.630735 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:21:15.630981 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:15.630805 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:21:15.810856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:15.810818 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:16.190123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:16.190084 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:16.631095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:16.631060 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:21:16.631280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:16.631127 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:21:17.383023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:17.382983 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:17.631705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:17.631669 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:21:17.631891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:17.631755 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:21:18.635323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:18.635288 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:18.635779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:18.635558 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:18.636201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:18.636182 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:21:18.908547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:18.908465 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:20.616720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:20.616683 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:22.262753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:22.262710 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:22.661616 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:22.608524 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:21:22.722597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:22.721739 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:22.744015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:22.743965 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:22.744357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:22.744327 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:22.851990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:22.744594 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:21:22.999737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:22.995107 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.063705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.063652 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.115687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.115436 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:21:23.154934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.154886 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 00:21:23.220600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.219124 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.497965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.496765 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:21:23.515968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.510813 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.691764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.691708 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.701432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.701401 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:21:23.748651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.723376 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:21:23.748651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.724461 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:21:23.759617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.750146 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.849471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.837044 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:23.902895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.850730 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 00:21:23.922564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.921796 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:21:23.981980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:23.981059 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:21:25.570425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.570393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:25.753089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.753053 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v"] Oct 30 00:21:25.758777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.758756 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.784155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.784122 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v"] Oct 30 00:21:25.832872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.832759 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.833045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.832918 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.833136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.833106 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4612bf-5599-477a-9abe-c2a15751d372-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.833275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.833256 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4612bf-5599-477a-9abe-c2a15751d372-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.848812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.848775 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.848974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.848831 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.848974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.848890 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggr5v\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-kube-api-access-ggr5v\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.849107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.849035 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.849107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.849075 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951094 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951318 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951143 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4612bf-5599-477a-9abe-c2a15751d372-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951318 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951219 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4612bf-5599-477a-9abe-c2a15751d372-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951318 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951273 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951318 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951292 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951330 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggr5v\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-kube-api-access-ggr5v\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951466 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951511 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951559 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951640 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951650 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.951912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.951836 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.952062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.952038 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.952128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.952096 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4612bf-5599-477a-9abe-c2a15751d372-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.953813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.953783 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.954032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.954013 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4612bf-5599-477a-9abe-c2a15751d372-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.975552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.975513 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:25.975708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:25.975626 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggr5v\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-kube-api-access-ggr5v\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:26.078989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:26.078955 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:26.413028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:26.413001 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v"] Oct 30 00:21:26.413412 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:26.413390 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4612bf_5599_477a_9abe_c2a15751d372.slice/crio-f15c62b287bd2af74d1eec21f2f002663ee1ba9b8ad4f83af04b2a8e7e0c23ad WatchSource:0}: Error finding container f15c62b287bd2af74d1eec21f2f002663ee1ba9b8ad4f83af04b2a8e7e0c23ad: Status 404 returned error can't find the container with id f15c62b287bd2af74d1eec21f2f002663ee1ba9b8ad4f83af04b2a8e7e0c23ad Oct 30 00:21:26.416073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:26.416038 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:26.416141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:26.416105 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:26.416141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:26.416135 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:27.112544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:27.111401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:27.409044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:27.408959 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" event={"ID":"eb4612bf-5599-477a-9abe-c2a15751d372","Type":"ContainerStarted","Data":"eb146ad4fb71b13f40c26536c680c9147ecc7a4a36aa39dd9abee3f801a4e1af"} Oct 30 00:21:27.409044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:27.409022 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" event={"ID":"eb4612bf-5599-477a-9abe-c2a15751d372","Type":"ContainerStarted","Data":"f15c62b287bd2af74d1eec21f2f002663ee1ba9b8ad4f83af04b2a8e7e0c23ad"} Oct 30 00:21:27.482587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:27.482505 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podStartSLOduration=2.482484501 podStartE2EDuration="2.482484501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:27.472360864 +0000 UTC m=+996.406429432" watchObservedRunningTime="2026-10-30 00:21:27.482484501 +0000 UTC m=+996.416553037" Oct 30 00:21:27.579768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:27.578839 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:28.079617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:28.079542 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:28.081167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:28.081102 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:21:28.081684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:28.081374 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:21:28.808279 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:28.808237 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:29.081417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:29.079887 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:21:29.081417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:29.079955 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:21:30.081016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:30.080976 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:21:30.081748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:30.081714 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:21:30.578513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:30.578477 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:31.087957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:31.087919 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:21:31.088611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:31.088557 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:21:32.080304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:32.080268 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:21:32.080562 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:32.080538 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:21:32.268618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:32.264720 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:33.079244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:33.079203 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:21:33.079399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:33.079258 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:21:33.871368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:33.871331 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:34.086123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:34.086090 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:34.086526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:34.086498 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:34.087441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:34.087416 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:21:35.383790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:35.383744 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:36.007370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:36.007329 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:36.758598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:36.756897 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:37.103028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:37.102992 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:37.105624 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:21:37.105586 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:21:37.345060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:37.345026 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:37.448952 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:37.448869 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:37.610455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:37.610419 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:38.329908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.329870 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j"] Oct 30 00:21:38.336506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.336476 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.357219 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.357192 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j"] Oct 30 00:21:38.365100 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.365075 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7417dc-openshift-default-dockercfg-j7dfg\"" Oct 30 00:21:38.432032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432000 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432120 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-data\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432165 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-podinfo\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432184 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istiod-ca-cert\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432204 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-certs\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432349 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-token\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432375 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-credential-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432459 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-envoy\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.432624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.432611 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5c8n\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-kube-api-access-l5c8n\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534339 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5c8n\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-kube-api-access-l5c8n\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534551 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534398 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534551 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534484 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-data\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534547 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-podinfo\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534602 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istiod-ca-cert\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534636 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-certs\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534842 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-token\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.534932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534893 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-credential-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.535027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534996 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-envoy\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.535027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.534894 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-data\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.535377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.535356 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.535482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.535453 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-certs\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.535695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.535667 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-credential-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.535881 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.535863 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istiod-ca-cert\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.537372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.537351 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-envoy\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.537372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.537358 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-podinfo\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.558934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.558902 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5c8n\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-kube-api-access-l5c8n\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.559131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.559108 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-token\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.652252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.652174 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:38.782612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.782545 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:38.978102 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:38.978067 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11a8c90_f95d_4aa1_a5ea_067a280a395a.slice/crio-d0953bb25658aa2046678a3d2f6dea40b55a7dee8c256faf5d1d59466b0b59f9 WatchSource:0}: Error finding container d0953bb25658aa2046678a3d2f6dea40b55a7dee8c256faf5d1d59466b0b59f9: Status 404 returned error can't find the container with id d0953bb25658aa2046678a3d2f6dea40b55a7dee8c256faf5d1d59466b0b59f9 Oct 30 00:21:38.979120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.979086 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j"] Oct 30 00:21:38.983208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.983164 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:38.983317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.983253 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:38.983317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:38.983282 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:21:39.569841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:39.562729 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" event={"ID":"a11a8c90-f95d-4aa1-a5ea-067a280a395a","Type":"ContainerStarted","Data":"266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de"} Oct 30 00:21:39.569841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:39.562805 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" event={"ID":"a11a8c90-f95d-4aa1-a5ea-067a280a395a","Type":"ContainerStarted","Data":"d0953bb25658aa2046678a3d2f6dea40b55a7dee8c256faf5d1d59466b0b59f9"} Oct 30 00:21:39.652494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:39.652450 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:39.673065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:39.667850 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podStartSLOduration=1.6678281780000002 podStartE2EDuration="1.667828178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:39.643775402 +0000 UTC m=+1008.577843972" watchObservedRunningTime="2026-10-30 00:21:39.667828178 +0000 UTC m=+1008.601896713" Oct 30 00:21:40.509961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:40.509926 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:40.615319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:40.615287 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:40.682815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:40.682784 4433 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:21:40.682990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:40.682871 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:21:40.745772 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:40.735541 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:41.423728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:41.420646 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:41.623862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:41.623819 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:41.659971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:41.659934 4433 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:21:41.660253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:41.660225 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:21:41.736656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:41.730857 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:42.180719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:42.180244 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:42.431754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:42.431667 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:42.545111 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:42.544999 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:42.709652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:42.709162 4433 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:21:42.709652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:42.709257 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:21:42.760121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:42.760040 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:42.874594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:42.874546 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:43.519241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.518779 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:43.619637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.619592 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:43.657597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.656043 4433 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:21:43.657597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.656105 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:21:43.710632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.710595 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28"] Oct 30 00:21:43.715687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.715663 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:43.735664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.735633 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28"] Oct 30 00:21:43.742051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.742020 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0aea47\"" Oct 30 00:21:43.742257 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.742238 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0aea47-dockercfg-gqgmx\"" Oct 30 00:21:43.831594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.828359 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x"] Oct 30 00:21:43.917595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.916596 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:43.917595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.916999 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4x6n\" (UniqueName: \"kubernetes.io/projected/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-kube-api-access-g4x6n\") pod \"payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:43.960600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.959492 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x"] Oct 30 00:21:43.960600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.959922 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:43.982916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:43.980205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:44.085659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.022121 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:44.085659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.022362 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4x6n\" (UniqueName: \"kubernetes.io/projected/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-kube-api-access-g4x6n\") pod \"payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:44.085659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.073430 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:44.158969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.149290 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcg4\" (UniqueName: \"kubernetes.io/projected/04e1172c-4907-4d1b-8276-fda61f96cdef-kube-api-access-vvcg4\") pod \"payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:44.158969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.149474 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1172c-4907-4d1b-8276-fda61f96cdef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:44.184612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.167863 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:44.201611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.185722 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4x6n\" (UniqueName: \"kubernetes.io/projected/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-kube-api-access-g4x6n\") pod \"payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:44.251244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.251205 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcg4\" (UniqueName: \"kubernetes.io/projected/04e1172c-4907-4d1b-8276-fda61f96cdef-kube-api-access-vvcg4\") pod \"payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:44.251437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.251348 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1172c-4907-4d1b-8276-fda61f96cdef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:44.252497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.252468 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1172c-4907-4d1b-8276-fda61f96cdef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:44.281201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.281166 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcg4\" (UniqueName: \"kubernetes.io/projected/04e1172c-4907-4d1b-8276-fda61f96cdef-kube-api-access-vvcg4\") pod \"payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:44.341515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.341421 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:44.548227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.547138 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:44.618997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.617115 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:44.719288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.716530 4433 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:21:44.719288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.716601 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:21:44.769896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.756933 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28"] Oct 30 00:21:44.792782 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:44.792744 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90dfe48_38f9_463a_aecc_887fa1d2c6e0.slice/crio-1f892d272fe18ea57e55b4929f7ccba8482139ec2d82f8e854a306e036368b3c WatchSource:0}: Error finding container 1f892d272fe18ea57e55b4929f7ccba8482139ec2d82f8e854a306e036368b3c: Status 404 returned error can't find the container with id 1f892d272fe18ea57e55b4929f7ccba8482139ec2d82f8e854a306e036368b3c Oct 30 00:21:44.825727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.825669 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" event={"ID":"c90dfe48-38f9-463a-aecc-887fa1d2c6e0","Type":"ContainerStarted","Data":"1f892d272fe18ea57e55b4929f7ccba8482139ec2d82f8e854a306e036368b3c"} Oct 30 00:21:44.947421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:44.947361 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x"] Oct 30 00:21:45.658228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.658194 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:45.658632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.658609 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:45.659264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.659244 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:21:45.837318 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.837277 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" event={"ID":"c90dfe48-38f9-463a-aecc-887fa1d2c6e0","Type":"ContainerStarted","Data":"655421f3bae80b61c90a32f8e05bd1ed34a0b011fc1a5c0ddcc06ed959bbf915"} Oct 30 00:21:45.837875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.837850 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:45.840066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.839893 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" event={"ID":"04e1172c-4907-4d1b-8276-fda61f96cdef","Type":"ContainerStarted","Data":"2e51be80b9be525fbba7b970cfabc49e5ffb5ca47b0706b6717cdbcfb5c0d4ff"} Oct 30 00:21:45.840066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.839969 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" event={"ID":"04e1172c-4907-4d1b-8276-fda61f96cdef","Type":"ContainerStarted","Data":"48fb72244a8d793f80e7c8d0fd2db19303ed44a1e54349b0f54860bea228b445"} Oct 30 00:21:45.840219 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.840129 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:45.886388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.886316 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" podStartSLOduration=2.8862951470000002 podStartE2EDuration="2.886295147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:45.884608582 +0000 UTC m=+1014.818677138" watchObservedRunningTime="2026-10-30 00:21:45.886295147 +0000 UTC m=+1014.820363685" Oct 30 00:21:45.930883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:45.930777 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" podStartSLOduration=2.930759787 podStartE2EDuration="2.930759787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:45.927530415 +0000 UTC m=+1014.861598950" watchObservedRunningTime="2026-10-30 00:21:45.930759787 +0000 UTC m=+1014.864828322" Oct 30 00:21:46.293437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:46.293355 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:47.030157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:47.030115 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:47.517842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:47.517803 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:47.772656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:47.771658 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:47.954159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:47.954110 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:48.065289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:48.065225 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:48.185419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:48.185382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:49.955523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:49.955004 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:50.058539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.058503 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:50.114418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.114387 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb"] Oct 30 00:21:50.119415 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.119390 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.138807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.138775 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb"] Oct 30 00:21:50.146945 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.146917 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-08dd33\"" Oct 30 00:21:50.147096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.146967 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-08dd33-dockercfg-dvzl6\"" Oct 30 00:21:50.178787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.178751 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l"] Oct 30 00:21:50.184992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.184964 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.199931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.199903 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l"] Oct 30 00:21:50.244165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.244059 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d224a3-01d7-49dc-893a-c40b6aeafb32-plugins-config-volume\") pod \"payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.244362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.244233 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8q28\" (UniqueName: \"kubernetes.io/projected/81d224a3-01d7-49dc-893a-c40b6aeafb32-kube-api-access-w8q28\") pod \"payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.345966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.345924 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d224a3-01d7-49dc-893a-c40b6aeafb32-plugins-config-volume\") pod \"payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.346161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.345979 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66607691-eb15-4d08-b2fc-edad9fff498f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.346161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.346034 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsbj\" (UniqueName: \"kubernetes.io/projected/66607691-eb15-4d08-b2fc-edad9fff498f-kube-api-access-wpsbj\") pod \"payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.346161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.346109 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8q28\" (UniqueName: \"kubernetes.io/projected/81d224a3-01d7-49dc-893a-c40b6aeafb32-kube-api-access-w8q28\") pod \"payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.348633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.347320 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d224a3-01d7-49dc-893a-c40b6aeafb32-plugins-config-volume\") pod \"payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.370771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.370743 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8q28\" (UniqueName: \"kubernetes.io/projected/81d224a3-01d7-49dc-893a-c40b6aeafb32-kube-api-access-w8q28\") pod \"payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.436488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.436438 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.460635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.447951 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsbj\" (UniqueName: \"kubernetes.io/projected/66607691-eb15-4d08-b2fc-edad9fff498f-kube-api-access-wpsbj\") pod \"payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.460635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.448385 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66607691-eb15-4d08-b2fc-edad9fff498f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.460635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.449297 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66607691-eb15-4d08-b2fc-edad9fff498f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.476722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.476659 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsbj\" (UniqueName: \"kubernetes.io/projected/66607691-eb15-4d08-b2fc-edad9fff498f-kube-api-access-wpsbj\") pod \"payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.506163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.506045 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:50.783563 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:50.783239 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d224a3_01d7_49dc_893a_c40b6aeafb32.slice/crio-bc3e3b07d6029f55d74d247fb3455651d15766dcbeda05f718fefda785aba804 WatchSource:0}: Error finding container bc3e3b07d6029f55d74d247fb3455651d15766dcbeda05f718fefda785aba804: Status 404 returned error can't find the container with id bc3e3b07d6029f55d74d247fb3455651d15766dcbeda05f718fefda785aba804 Oct 30 00:21:50.785558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.785130 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb"] Oct 30 00:21:50.887532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.887484 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l"] Oct 30 00:21:50.900375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.900331 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" event={"ID":"81d224a3-01d7-49dc-893a-c40b6aeafb32","Type":"ContainerStarted","Data":"bc3e3b07d6029f55d74d247fb3455651d15766dcbeda05f718fefda785aba804"} Oct 30 00:21:50.901344 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.901316 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:21:50.934440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.933990 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:50.951016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:50.950956 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" podStartSLOduration=0.95093926 podStartE2EDuration="950.93926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:50.950270054 +0000 UTC m=+1019.884338589" watchObservedRunningTime="2026-10-30 00:21:50.95093926 +0000 UTC m=+1019.885007796" Oct 30 00:21:51.037815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.037729 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:51.122009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.121975 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj"] Oct 30 00:21:51.126740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.126718 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.142808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.142776 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f52ea1\"" Oct 30 00:21:51.142932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.142825 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f52ea1-dockercfg-86cvs\"" Oct 30 00:21:51.144738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.144714 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj"] Oct 30 00:21:51.218788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.218756 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx"] Oct 30 00:21:51.237305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.237001 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx"] Oct 30 00:21:51.237540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.237260 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.295897 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.295803 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87wq\" (UniqueName: \"kubernetes.io/projected/f32e80dd-fc56-43e3-9b8d-33934d72248d-kube-api-access-b87wq\") pod \"payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.301757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.301720 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607d678f-e613-4400-b51a-1cb4b41c9a12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.305846 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.305817 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32e80dd-fc56-43e3-9b8d-33934d72248d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.306210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.306190 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpj8g\" (UniqueName: \"kubernetes.io/projected/607d678f-e613-4400-b51a-1cb4b41c9a12-kube-api-access-wpj8g\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.409017 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.408970 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87wq\" (UniqueName: \"kubernetes.io/projected/f32e80dd-fc56-43e3-9b8d-33934d72248d-kube-api-access-b87wq\") pod \"payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.409218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.409153 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607d678f-e613-4400-b51a-1cb4b41c9a12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.409218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.409201 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32e80dd-fc56-43e3-9b8d-33934d72248d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.409340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.409259 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpj8g\" (UniqueName: \"kubernetes.io/projected/607d678f-e613-4400-b51a-1cb4b41c9a12-kube-api-access-wpj8g\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.424641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.424609 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f52ea1\"" Oct 30 00:21:51.432833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.431810 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32e80dd-fc56-43e3-9b8d-33934d72248d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.432833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.432667 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607d678f-e613-4400-b51a-1cb4b41c9a12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.435782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.435752 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpj8g\" (UniqueName: \"kubernetes.io/projected/607d678f-e613-4400-b51a-1cb4b41c9a12-kube-api-access-wpj8g\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.440340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.440313 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87wq\" (UniqueName: \"kubernetes.io/projected/f32e80dd-fc56-43e3-9b8d-33934d72248d-kube-api-access-b87wq\") pod \"payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.456833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.456798 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f52ea1-dockercfg-86cvs\"" Oct 30 00:21:51.463024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.462988 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.570117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.570081 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:51.777105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.777058 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:51.789149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.789096 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj"] Oct 30 00:21:51.789589 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:51.789549 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607d678f_e613_4400_b51a_1cb4b41c9a12.slice/crio-0f64c780b9229961758d759ce4ab7f7c3f3e4f30926f753d9de19fd339c1f74a WatchSource:0}: Error finding container 0f64c780b9229961758d759ce4ab7f7c3f3e4f30926f753d9de19fd339c1f74a: Status 404 returned error can't find the container with id 0f64c780b9229961758d759ce4ab7f7c3f3e4f30926f753d9de19fd339c1f74a Oct 30 00:21:51.910025 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.909974 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" event={"ID":"66607691-eb15-4d08-b2fc-edad9fff498f","Type":"ContainerStarted","Data":"a61f7ad20f8d28bf2a1498825b5a49867a54ae3a491fba9f71ecccd543d19580"} Oct 30 00:21:51.910953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.910057 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:21:51.910953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.910089 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" event={"ID":"66607691-eb15-4d08-b2fc-edad9fff498f","Type":"ContainerStarted","Data":"dc984adc0f5a2729def4325891cb5a33492f6f7fa4037e0a32e4096c7f7d9553"} Oct 30 00:21:51.914664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.914624 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" event={"ID":"607d678f-e613-4400-b51a-1cb4b41c9a12","Type":"ContainerStarted","Data":"f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa"} Oct 30 00:21:51.914809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.914708 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:21:51.914809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.914739 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" event={"ID":"607d678f-e613-4400-b51a-1cb4b41c9a12","Type":"ContainerStarted","Data":"0f64c780b9229961758d759ce4ab7f7c3f3e4f30926f753d9de19fd339c1f74a"} Oct 30 00:21:51.917452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.917425 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" event={"ID":"81d224a3-01d7-49dc-893a-c40b6aeafb32","Type":"ContainerStarted","Data":"6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b"} Oct 30 00:21:51.926262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.926234 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx"] Oct 30 00:21:51.926633 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:51.926563 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32e80dd_fc56_43e3_9b8d_33934d72248d.slice/crio-dcd277457cd46f6e555c167b7baab25df365ada31f90aa219a82e079a5294b7d WatchSource:0}: Error finding container dcd277457cd46f6e555c167b7baab25df365ada31f90aa219a82e079a5294b7d: Status 404 returned error can't find the container with id dcd277457cd46f6e555c167b7baab25df365ada31f90aa219a82e079a5294b7d Oct 30 00:21:51.954546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.954496 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" podStartSLOduration=1.9544814179999999 podStartE2EDuration="1.954481418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:51.951021724 +0000 UTC m=+1020.885090259" watchObservedRunningTime="2026-10-30 00:21:51.954481418 +0000 UTC m=+1020.888549953" Oct 30 00:21:51.999539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:51.999482 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" podStartSLOduration=0.999466309 podStartE2EDuration="999.466309ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:51.996546875 +0000 UTC m=+1020.930615430" watchObservedRunningTime="2026-10-30 00:21:51.999466309 +0000 UTC m=+1020.933534843" Oct 30 00:21:52.927751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:52.927698 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" event={"ID":"f32e80dd-fc56-43e3-9b8d-33934d72248d","Type":"ContainerStarted","Data":"2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72"} Oct 30 00:21:52.928194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:52.927786 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" event={"ID":"f32e80dd-fc56-43e3-9b8d-33934d72248d","Type":"ContainerStarted","Data":"dcd277457cd46f6e555c167b7baab25df365ada31f90aa219a82e079a5294b7d"} Oct 30 00:21:52.972566 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:52.972512 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" podStartSLOduration=1.97249438 podStartE2EDuration="1.97249438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:52.968934203 +0000 UTC m=+1021.903002758" watchObservedRunningTime="2026-10-30 00:21:52.97249438 +0000 UTC m=+1021.906562926" Oct 30 00:21:53.264052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:53.263960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:53.372802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:53.372766 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:53.478645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:53.478550 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:53.937056 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:53.937018 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:21:55.007245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.007210 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:55.123298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.123260 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:55.160626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.160593 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6"] Oct 30 00:21:55.165405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.165377 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.185565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.185536 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6"] Oct 30 00:21:55.190409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.190380 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c34304\"" Oct 30 00:21:55.190638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.190496 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c34304-dockercfg-zl4nk\"" Oct 30 00:21:55.261215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.261144 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w"] Oct 30 00:21:55.269388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.269357 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.285724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.285691 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:55.286238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.286212 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w"] Oct 30 00:21:55.287758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.287731 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.287916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.287896 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64zr\" (UniqueName: \"kubernetes.io/projected/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-kube-api-access-b64zr\") pod \"payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.396462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.389793 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64zr\" (UniqueName: \"kubernetes.io/projected/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-kube-api-access-b64zr\") pod \"payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.396462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.390137 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxmw\" (UniqueName: \"kubernetes.io/projected/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-kube-api-access-sfxmw\") pod \"payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.396462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.390239 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.396462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.390301 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.396462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.391646 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.437593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.427561 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64zr\" (UniqueName: \"kubernetes.io/projected/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-kube-api-access-b64zr\") pod \"payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.482596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.481924 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:55.503593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.496149 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxmw\" (UniqueName: \"kubernetes.io/projected/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-kube-api-access-sfxmw\") pod \"payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.503593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.496306 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.503593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.497317 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.544281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.544249 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxmw\" (UniqueName: \"kubernetes.io/projected/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-kube-api-access-sfxmw\") pod \"payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.598598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.594023 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:55.911134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.911099 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6"] Oct 30 00:21:55.917300 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:55.917268 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ebded0_f3c8_40b9_89d2_5c4e1f3e2074.slice/crio-71a99fff076f6c0a86d800d84d5f3a9225f3fc7af76afa974c2ffc811c167337 WatchSource:0}: Error finding container 71a99fff076f6c0a86d800d84d5f3a9225f3fc7af76afa974c2ffc811c167337: Status 404 returned error can't find the container with id 71a99fff076f6c0a86d800d84d5f3a9225f3fc7af76afa974c2ffc811c167337 Oct 30 00:21:55.929776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.929749 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w"] Oct 30 00:21:55.934642 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:21:55.934613 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c4887a_af7b_4fcc_a8f9_d5b3198da80a.slice/crio-7bcfd86fa9704681181b3491db8c1efac8b19668d60ad11966e494d4a8d5a875 WatchSource:0}: Error finding container 7bcfd86fa9704681181b3491db8c1efac8b19668d60ad11966e494d4a8d5a875: Status 404 returned error can't find the container with id 7bcfd86fa9704681181b3491db8c1efac8b19668d60ad11966e494d4a8d5a875 Oct 30 00:21:55.954333 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.954306 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" event={"ID":"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074","Type":"ContainerStarted","Data":"71a99fff076f6c0a86d800d84d5f3a9225f3fc7af76afa974c2ffc811c167337"} Oct 30 00:21:55.955627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:55.955597 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" event={"ID":"65c4887a-af7b-4fcc-a8f9-d5b3198da80a","Type":"ContainerStarted","Data":"7bcfd86fa9704681181b3491db8c1efac8b19668d60ad11966e494d4a8d5a875"} Oct 30 00:21:56.855829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:56.855796 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:21:56.858758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:56.858731 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:21:56.897739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:56.897693 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:57.005654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:57.005610 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" event={"ID":"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074","Type":"ContainerStarted","Data":"44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa"} Oct 30 00:21:57.005850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:57.005710 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:21:57.007783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:57.007753 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" event={"ID":"65c4887a-af7b-4fcc-a8f9-d5b3198da80a","Type":"ContainerStarted","Data":"84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6"} Oct 30 00:21:57.007983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:57.007858 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:21:57.067597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:57.056357 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" podStartSLOduration=2.056339929 podStartE2EDuration="2.056339929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:57.046843083 +0000 UTC m=+1025.980911617" watchObservedRunningTime="2026-10-30 00:21:57.056339929 +0000 UTC m=+1025.990408465" Oct 30 00:21:57.107693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:57.096049 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" podStartSLOduration=2.096028775 podStartE2EDuration="2.096028775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:57.094747859 +0000 UTC m=+1026.028816395" watchObservedRunningTime="2026-10-30 00:21:57.096028775 +0000 UTC m=+1026.030097314" Oct 30 00:21:58.594593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:58.593485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:21:59.515598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.509649 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:59.727603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.715665 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 00:21:59.727603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.718297 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:59.749599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.745689 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:59.762606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.760659 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:59.830767 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.828647 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:59.927947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.927708 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:59.996600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:21:59.984763 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.048247 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.047672 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.086683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.084780 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.086683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.085628 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:22:00.091608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.090659 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 00:22:00.207606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.206721 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:22:00.722595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.716668 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.722595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.717445 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:22:00.722595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.718531 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.723110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.723022 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:22:00.745557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.745418 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.781075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.754210 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.782835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.781390 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:22:00.908404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.865825 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:00.950646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.934035 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.950646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:00.934347 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:22:01.139656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.121750 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.139656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.134113 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:22:01.139656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.135745 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:22:01.139656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.135810 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:22:01.229623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.221099 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.237901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.230716 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:22:01.237901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.231630 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.319592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.310136 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:22:01.319592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.310194 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:22:01.320647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.320133 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:22:01.327528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.327120 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:22:01.395650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.394858 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.484606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.484426 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:22:01.514785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.514699 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:22:01.546126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.546049 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:22:01.606048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.606017 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:22:01.616413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:01.616388 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:22:02.856121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:02.856082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:02.908601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:02.907344 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng"] Oct 30 00:22:02.971598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:02.966272 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:02.993606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:02.991773 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:03.020610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.013953 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0aea47-serving-cert\"" Oct 30 00:22:03.020610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.017421 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:22:03.136896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.075751 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:22:03.136896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.106957 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-metrics-certs\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.136896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.107158 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-maas-api-tls\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.136896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.107242 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c76eba-4a75-4417-a6f4-21b297c54ac0-openshift-service-ca\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.136896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.107377 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkpv\" (UniqueName: \"kubernetes.io/projected/d9c76eba-4a75-4417-a6f4-21b297c54ac0-kube-api-access-tjkpv\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.160606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.159148 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:03.160606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.159878 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 00:22:03.377765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.377716 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.446651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.442722 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-metrics-certs\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.460695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.460651 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:22:03.464594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.461013 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-maas-api-tls\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.561150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.472356 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:22:03.561408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.561388 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng"] Oct 30 00:22:03.561556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.490821 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:22:03.566595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.561866 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c76eba-4a75-4417-a6f4-21b297c54ac0-openshift-service-ca\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.566595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.559710 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 00:22:03.570622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.490858 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-phppj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:22:03.570781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.570759 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.570909 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.559796 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:22:03.592593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.570999 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.592801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.559747 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:22:03.593990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.561128 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c76eba-4a75-4417-a6f4-21b297c54ac0-openshift-service-ca\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.594259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.594235 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkpv\" (UniqueName: \"kubernetes.io/projected/d9c76eba-4a75-4417-a6f4-21b297c54ac0-kube-api-access-tjkpv\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.618593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.490790 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:22:03.618812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.618780 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.627469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.619948 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.627469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.621116 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:22:03.664602 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.661546 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-maas-api-tls\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.809640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.784336 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkpv\" (UniqueName: \"kubernetes.io/projected/d9c76eba-4a75-4417-a6f4-21b297c54ac0-kube-api-access-tjkpv\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.809884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.798410 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-metrics-certs\") pod \"maas-api-e2e-api-b-0aea47-67c95967b5-zknng\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:03.810016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.798813 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.827478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.827433 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.844728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:03.842094 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:04.400619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.400529 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:22:04.409597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.407605 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:22:04.452598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.449043 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cj97z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:22:04.452598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.449102 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cj97z" podUID="e0d5fcfc-fadc-44e6-9b61-986bc7395aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.468501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.460904 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:22:04.468501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.462306 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:22:04.563741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.562546 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:22:04.564985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.564054 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:22:04.592341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.588323 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:22:04.659671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.618128 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.659671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.621273 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:22:04.659671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.623894 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.659671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.624162 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.659671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.624399 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:22:04.659671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.624445 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:22:04.772979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.772172 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:22:04.858776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.858685 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.859652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.859630 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:22:04.865295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.865277 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:22:04.901228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.900635 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:22:04.962437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:04.962404 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:22:05.046103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:05.043845 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:22:05.077443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:05.077380 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:22:05.101805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:05.100523 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:22:05.102249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:05.102148 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:05.249646 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:22:05.249609 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c76eba_4a75_4417_a6f4_21b297c54ac0.slice/crio-2058e11c27a9577c7b68045ba3bf8a10a1288e2793f5cbdf65409c500456675f WatchSource:0}: Error finding container 2058e11c27a9577c7b68045ba3bf8a10a1288e2793f5cbdf65409c500456675f: Status 404 returned error can't find the container with id 2058e11c27a9577c7b68045ba3bf8a10a1288e2793f5cbdf65409c500456675f Oct 30 00:22:05.251199 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:05.251178 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:22:05.253766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:05.252385 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng"] Oct 30 00:22:06.250484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:06.250453 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" event={"ID":"d9c76eba-4a75-4417-a6f4-21b297c54ac0","Type":"ContainerStarted","Data":"799bdc1e85f824e02470f47312ca6ba216caa931045ab6f3daa280bd7b7f9a91"} Oct 30 00:22:06.250887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:06.250505 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" event={"ID":"d9c76eba-4a75-4417-a6f4-21b297c54ac0","Type":"ContainerStarted","Data":"2058e11c27a9577c7b68045ba3bf8a10a1288e2793f5cbdf65409c500456675f"} Oct 30 00:22:06.250887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:06.250617 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:06.298043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:06.297982 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" podStartSLOduration=3.837246957 podStartE2EDuration="4.297967932s" totalImagesPullingTime="460.720975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:22:06.294991851 +0000 UTC m=+1035.229060386" watchObservedRunningTime="2026-10-30 00:22:06.297967932 +0000 UTC m=+1035.232036467" Oct 30 00:22:06.605323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:06.605287 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:07.105811 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:22:07.105775 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:22:08.037463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:08.037428 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:22:08.038066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:08.038048 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:22:08.086190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:08.086157 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:09.485915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:09.485869 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:10.833881 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.833836 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm"] Oct 30 00:22:10.838829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.838805 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:10.859199 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.859164 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm"] Oct 30 00:22:10.865661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.865637 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-08dd33-serving-cert\"" Oct 30 00:22:10.913699 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.913655 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:10.957561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.957521 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-metrics-certs\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:10.957759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.957654 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-maas-api-tls\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:10.958011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.957986 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvxw\" (UniqueName: \"kubernetes.io/projected/5ee60444-e0e5-46cb-81fc-9522041a5b22-kube-api-access-sjvxw\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:10.958055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:10.958035 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee60444-e0e5-46cb-81fc-9522041a5b22-openshift-service-ca\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.016216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.016169 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:11.060740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.060387 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvxw\" (UniqueName: \"kubernetes.io/projected/5ee60444-e0e5-46cb-81fc-9522041a5b22-kube-api-access-sjvxw\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.060740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.060474 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee60444-e0e5-46cb-81fc-9522041a5b22-openshift-service-ca\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.060740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.060611 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-metrics-certs\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.060740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.060722 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-maas-api-tls\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.067518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.065543 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee60444-e0e5-46cb-81fc-9522041a5b22-openshift-service-ca\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.082668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.082265 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-maas-api-tls\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.082668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.082389 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-metrics-certs\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.097241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.097162 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvxw\" (UniqueName: \"kubernetes.io/projected/5ee60444-e0e5-46cb-81fc-9522041a5b22-kube-api-access-sjvxw\") pod \"maas-api-e2e-api-b-08dd33-769fd9b55-9gffm\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.121459 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.121420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:11.159190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.159165 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:11.487636 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:22:11.487568 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee60444_e0e5_46cb_81fc_9522041a5b22.slice/crio-36c93e6b23289d91d0f3ab19b06cdb4391bde48a0de17897f55321f2d1858cec WatchSource:0}: Error finding container 36c93e6b23289d91d0f3ab19b06cdb4391bde48a0de17897f55321f2d1858cec: Status 404 returned error can't find the container with id 36c93e6b23289d91d0f3ab19b06cdb4391bde48a0de17897f55321f2d1858cec Oct 30 00:22:11.487791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:11.487765 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm"] Oct 30 00:22:12.271639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.269208 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:22:12.309053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.309011 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" event={"ID":"5ee60444-e0e5-46cb-81fc-9522041a5b22","Type":"ContainerStarted","Data":"86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637"} Oct 30 00:22:12.309263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.309084 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" event={"ID":"5ee60444-e0e5-46cb-81fc-9522041a5b22","Type":"ContainerStarted","Data":"36c93e6b23289d91d0f3ab19b06cdb4391bde48a0de17897f55321f2d1858cec"} Oct 30 00:22:12.309263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.309120 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:12.364185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.364128 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" podStartSLOduration=2.082418778 podStartE2EDuration="2.364113915s" totalImagesPullingTime="281.695137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:22:12.359961214 +0000 UTC m=+1041.294029798" watchObservedRunningTime="2026-10-30 00:22:12.364113915 +0000 UTC m=+1041.298182450" Oct 30 00:22:12.657626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.652656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:12.759119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.759072 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:12.860271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:12.860236 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:14.442038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:14.441937 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:15.916193 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:15.916157 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:17.502925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:17.502891 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:18.323515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:18.323486 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:22:19.052954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:19.052912 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:19.156751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:19.156707 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:19.259998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:19.259943 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:20.685155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:20.685111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:22.104504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:22.104470 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:23.601248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:23.601204 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:25.124422 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:25.124389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:26.764874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:26.764839 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:27.083820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:27.083788 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:27.196920 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:27.196069 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:27.982155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:27.982118 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4"] Oct 30 00:22:27.982618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:27.982394 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" podUID="b619fe16-234d-4a85-b829-39325260091d" containerName="maas-api" containerID="cri-o://9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5" gracePeriod=30 Oct 30 00:22:28.036276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.036245 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:28.153089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.153048 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" podUID="b619fe16-234d-4a85-b829-39325260091d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.86:8443/health\": dial tcp 10.134.0.86:8443: connect: connection refused" Oct 30 00:22:28.384630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.378274 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:22:28.437777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.437742 4433 generic.go:356] "Generic (PLEG): container finished" podID="b619fe16-234d-4a85-b829-39325260091d" containerID="9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5" exitCode=0 Oct 30 00:22:28.437980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.437815 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" Oct 30 00:22:28.437980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.437824 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" event={"ID":"b619fe16-234d-4a85-b829-39325260091d","Type":"ContainerDied","Data":"9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5"} Oct 30 00:22:28.437980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.437880 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4" event={"ID":"b619fe16-234d-4a85-b829-39325260091d","Type":"ContainerDied","Data":"a237abeb6a34799d29d13a1e3d777037898b71c7af7bc570def30b02fcb15d83"} Oct 30 00:22:28.437980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.437909 4433 scope.go:122] "RemoveContainer" containerID="9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5" Oct 30 00:22:28.440289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.440264 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b619fe16-234d-4a85-b829-39325260091d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b619fe16-234d-4a85-b829-39325260091d-openshift-service-ca\") pod \"b619fe16-234d-4a85-b829-39325260091d\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " Oct 30 00:22:28.440433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.440363 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-maas-api-tls\") pod \"b619fe16-234d-4a85-b829-39325260091d\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " Oct 30 00:22:28.440689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.440667 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b619fe16-234d-4a85-b829-39325260091d-openshift-service-ca" pod "b619fe16-234d-4a85-b829-39325260091d" (UID: "b619fe16-234d-4a85-b829-39325260091d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:28.440792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.440706 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-metrics-certs\") pod \"b619fe16-234d-4a85-b829-39325260091d\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " Oct 30 00:22:28.440792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.440743 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b619fe16-234d-4a85-b829-39325260091d-kube-api-access-rzzzk\" (UniqueName: \"kubernetes.io/projected/b619fe16-234d-4a85-b829-39325260091d-kube-api-access-rzzzk\") pod \"b619fe16-234d-4a85-b829-39325260091d\" (UID: \"b619fe16-234d-4a85-b829-39325260091d\") " Oct 30 00:22:28.441480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.441455 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b619fe16-234d-4a85-b829-39325260091d-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:28.443227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.443182 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-metrics-certs" pod "b619fe16-234d-4a85-b829-39325260091d" (UID: "b619fe16-234d-4a85-b829-39325260091d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:22:28.443346 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.443287 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-maas-api-tls" pod "b619fe16-234d-4a85-b829-39325260091d" (UID: "b619fe16-234d-4a85-b829-39325260091d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:22:28.443649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.443630 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b619fe16-234d-4a85-b829-39325260091d-kube-api-access-rzzzk" pod "b619fe16-234d-4a85-b829-39325260091d" (UID: "b619fe16-234d-4a85-b829-39325260091d"). InnerVolumeSpecName "kube-api-access-rzzzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:28.451278 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.451256 4433 scope.go:122] "RemoveContainer" containerID="9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5" Oct 30 00:22:28.451686 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:22:28.451560 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5\": container with ID starting with 9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5 not found: ID does not exist" containerID="9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5" Oct 30 00:22:28.451686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.451660 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5"} err="failed to get container status \"9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5\": rpc error: code = NotFound desc = could not find container \"9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5\": container with ID starting with 9ad2b2445342c219602cf47e54cf0f00998824841c3c92ea18b143eb03ba84a5 not found: ID does not exist" Oct 30 00:22:28.552642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.549315 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:28.552642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.551677 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzzzk\" (UniqueName: \"kubernetes.io/projected/b619fe16-234d-4a85-b829-39325260091d-kube-api-access-rzzzk\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:28.552642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.551706 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b619fe16-234d-4a85-b829-39325260091d-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:28.633942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.633904 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:28.739114 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.739049 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:28.793765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.793734 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4"] Oct 30 00:22:28.798376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.798350 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3fae6d-8856c7455-v64f4"] Oct 30 00:22:28.853215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:28.852901 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:29.185921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:29.185826 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b619fe16-234d-4a85-b829-39325260091d" path="/var/lib/kubelet/pods/b619fe16-234d-4a85-b829-39325260091d/volumes" Oct 30 00:22:29.266121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:29.266088 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:30.480298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:30.480263 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:32.094261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:32.094221 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:32.209225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:32.209193 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:33.681361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:33.681321 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:35.117131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:35.117090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:35.225291 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:35.225257 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:35.326138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:35.326103 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:35.468375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:35.468286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:36.925802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:36.925767 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:37.106638 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:22:37.106603 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:22:37.464278 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:37.464237 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:38.519126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:38.519090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:40.305301 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:40.305259 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:42.505764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:42.505732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:43.815240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:43.815202 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:44.217551 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:44.217449 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx"] Oct 30 00:22:44.252432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:44.252400 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj"] Oct 30 00:22:44.356182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:44.356149 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:44.465708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:44.465443 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:44.565906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:44.565870 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:46.065975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:46.065938 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:46.348990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:46.348904 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:46.451720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:46.451685 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:47.873615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:47.873531 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:49.218159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.218124 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" podUID="f32e80dd-fc56-43e3-9b8d-33934d72248d" containerName="payload-processing" containerID="cri-o://2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72" gracePeriod=25 Oct 30 00:22:49.252971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.252929 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" podUID="607d678f-e613-4400-b51a-1cb4b41c9a12" containerName="payload-pre-processing" containerID="cri-o://f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa" gracePeriod=25 Oct 30 00:22:49.375290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.375197 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:49.620019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.619647 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:22:49.646133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.646096 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:22:49.759233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759203 4433 generic.go:356] "Generic (PLEG): container finished" podID="f32e80dd-fc56-43e3-9b8d-33934d72248d" containerID="2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72" exitCode=0 Oct 30 00:22:49.759421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759256 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" Oct 30 00:22:49.759421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759293 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" event={"ID":"f32e80dd-fc56-43e3-9b8d-33934d72248d","Type":"ContainerDied","Data":"2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72"} Oct 30 00:22:49.759421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759350 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx" event={"ID":"f32e80dd-fc56-43e3-9b8d-33934d72248d","Type":"ContainerDied","Data":"dcd277457cd46f6e555c167b7baab25df365ada31f90aa219a82e079a5294b7d"} Oct 30 00:22:49.759421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759351 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32e80dd-fc56-43e3-9b8d-33934d72248d-kube-api-access-b87wq\" (UniqueName: \"kubernetes.io/projected/f32e80dd-fc56-43e3-9b8d-33934d72248d-kube-api-access-b87wq\") pod \"f32e80dd-fc56-43e3-9b8d-33934d72248d\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " Oct 30 00:22:49.759421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759378 4433 scope.go:122] "RemoveContainer" containerID="2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72" Oct 30 00:22:49.759655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759472 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607d678f-e613-4400-b51a-1cb4b41c9a12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607d678f-e613-4400-b51a-1cb4b41c9a12-plugins-config-volume\") pod \"607d678f-e613-4400-b51a-1cb4b41c9a12\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " Oct 30 00:22:49.759705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759688 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32e80dd-fc56-43e3-9b8d-33934d72248d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32e80dd-fc56-43e3-9b8d-33934d72248d-plugins-config-volume\") pod \"f32e80dd-fc56-43e3-9b8d-33934d72248d\" (UID: \"f32e80dd-fc56-43e3-9b8d-33934d72248d\") " Oct 30 00:22:49.759862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759838 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607d678f-e613-4400-b51a-1cb4b41c9a12-plugins-config-volume" pod "607d678f-e613-4400-b51a-1cb4b41c9a12" (UID: "607d678f-e613-4400-b51a-1cb4b41c9a12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:49.760154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.760102 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32e80dd-fc56-43e3-9b8d-33934d72248d-plugins-config-volume" pod "f32e80dd-fc56-43e3-9b8d-33934d72248d" (UID: "f32e80dd-fc56-43e3-9b8d-33934d72248d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:49.760154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.759894 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607d678f-e613-4400-b51a-1cb4b41c9a12-kube-api-access-wpj8g\" (UniqueName: \"kubernetes.io/projected/607d678f-e613-4400-b51a-1cb4b41c9a12-kube-api-access-wpj8g\") pod \"607d678f-e613-4400-b51a-1cb4b41c9a12\" (UID: \"607d678f-e613-4400-b51a-1cb4b41c9a12\") " Oct 30 00:22:49.761083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.761063 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32e80dd-fc56-43e3-9b8d-33934d72248d-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:49.761083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.761089 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607d678f-e613-4400-b51a-1cb4b41c9a12-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:49.761630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.761604 4433 generic.go:356] "Generic (PLEG): container finished" podID="607d678f-e613-4400-b51a-1cb4b41c9a12" containerID="f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa" exitCode=0 Oct 30 00:22:49.761737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.761661 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" event={"ID":"607d678f-e613-4400-b51a-1cb4b41c9a12","Type":"ContainerDied","Data":"f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa"} Oct 30 00:22:49.761737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.761719 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" event={"ID":"607d678f-e613-4400-b51a-1cb4b41c9a12","Type":"ContainerDied","Data":"0f64c780b9229961758d759ce4ab7f7c3f3e4f30926f753d9de19fd339c1f74a"} Oct 30 00:22:49.761861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.761790 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj" Oct 30 00:22:49.762158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.762137 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607d678f-e613-4400-b51a-1cb4b41c9a12-kube-api-access-wpj8g" pod "607d678f-e613-4400-b51a-1cb4b41c9a12" (UID: "607d678f-e613-4400-b51a-1cb4b41c9a12"). InnerVolumeSpecName "kube-api-access-wpj8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:49.762228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.762202 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32e80dd-fc56-43e3-9b8d-33934d72248d-kube-api-access-b87wq" pod "f32e80dd-fc56-43e3-9b8d-33934d72248d" (UID: "f32e80dd-fc56-43e3-9b8d-33934d72248d"). InnerVolumeSpecName "kube-api-access-b87wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:49.770692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.770657 4433 scope.go:122] "RemoveContainer" containerID="2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72" Oct 30 00:22:49.770946 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:22:49.770928 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72\": container with ID starting with 2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72 not found: ID does not exist" containerID="2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72" Oct 30 00:22:49.771011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.770952 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72"} err="failed to get container status \"2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72\": rpc error: code = NotFound desc = could not find container \"2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72\": container with ID starting with 2f5acaca35001f5f394076c1a170035cb32891ea4c5b12cf8cad20ddb418ed72 not found: ID does not exist" Oct 30 00:22:49.771011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.770966 4433 scope.go:122] "RemoveContainer" containerID="f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa" Oct 30 00:22:49.781511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.781494 4433 scope.go:122] "RemoveContainer" containerID="f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa" Oct 30 00:22:49.781818 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:22:49.781797 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa\": container with ID starting with f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa not found: ID does not exist" containerID="f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa" Oct 30 00:22:49.781901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.781827 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa"} err="failed to get container status \"f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa\": rpc error: code = NotFound desc = could not find container \"f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa\": container with ID starting with f22612aa49046a90e4b2eac8da4c8cb4db6e4ddeeeade8a509355950f1e293fa not found: ID does not exist" Oct 30 00:22:49.863353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.863322 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpj8g\" (UniqueName: \"kubernetes.io/projected/607d678f-e613-4400-b51a-1cb4b41c9a12-kube-api-access-wpj8g\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:49.863353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:49.863354 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87wq\" (UniqueName: \"kubernetes.io/projected/f32e80dd-fc56-43e3-9b8d-33934d72248d-kube-api-access-b87wq\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:50.112660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:50.112631 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx"] Oct 30 00:22:50.126005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:50.125973 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-784fd4bf48-5hlzx"] Oct 30 00:22:50.151929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:50.151892 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj"] Oct 30 00:22:50.156857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:50.156831 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-65f577b667-sc5kj"] Oct 30 00:22:50.941808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:50.941776 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:51.182462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:51.182429 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607d678f-e613-4400-b51a-1cb4b41c9a12" path="/var/lib/kubelet/pods/607d678f-e613-4400-b51a-1cb4b41c9a12/volumes" Oct 30 00:22:51.183015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:51.182998 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32e80dd-fc56-43e3-9b8d-33934d72248d" path="/var/lib/kubelet/pods/f32e80dd-fc56-43e3-9b8d-33934d72248d/volumes" Oct 30 00:22:52.351324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:52.351278 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:53.894734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:53.894691 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:54.016035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:54.016007 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w"] Oct 30 00:22:54.056773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:54.056735 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6"] Oct 30 00:22:54.146507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:54.146430 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:54.248204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:54.248170 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:54.351795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:54.351763 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:55.487586 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:55.487535 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:57.028230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:57.028195 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:57.191273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:57.191235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:57.591194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:57.591162 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:58.767288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:58.767248 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:22:59.016955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.016905 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" podUID="65c4887a-af7b-4fcc-a8f9-d5b3198da80a" containerName="payload-processing" containerID="cri-o://84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6" gracePeriod=25 Oct 30 00:22:59.058249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.058209 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" podUID="64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" containerName="payload-pre-processing" containerID="cri-o://44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa" gracePeriod=25 Oct 30 00:22:59.150300 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.150268 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr"] Oct 30 00:22:59.151108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151091 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b619fe16-234d-4a85-b829-39325260091d" containerName="maas-api" Oct 30 00:22:59.151166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151110 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b619fe16-234d-4a85-b829-39325260091d" containerName="maas-api" Oct 30 00:22:59.151166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151140 4433 cpu_manager.go:414] "containerMap: removing container" podUID="607d678f-e613-4400-b51a-1cb4b41c9a12" containerName="payload-pre-processing" Oct 30 00:22:59.151166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151148 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607d678f-e613-4400-b51a-1cb4b41c9a12" containerName="payload-pre-processing" Oct 30 00:22:59.151166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151161 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f32e80dd-fc56-43e3-9b8d-33934d72248d" containerName="payload-processing" Oct 30 00:22:59.151319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151170 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32e80dd-fc56-43e3-9b8d-33934d72248d" containerName="payload-processing" Oct 30 00:22:59.151421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151404 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b619fe16-234d-4a85-b829-39325260091d" containerName="maas-api" Oct 30 00:22:59.151464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151429 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="607d678f-e613-4400-b51a-1cb4b41c9a12" containerName="payload-pre-processing" Oct 30 00:22:59.151464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.151439 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32e80dd-fc56-43e3-9b8d-33934d72248d" containerName="payload-processing" Oct 30 00:22:59.155254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.155235 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.171084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.171065 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f52ea1-openshift-default-dockercfg-6zrs8\"" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.331559 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.331691 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.331756 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.331824 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.331863 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.331907 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.331947 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.332026 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwb4\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-kube-api-access-hdwb4\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.332227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.332061 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.362681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.362648 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:22:59.396094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.396069 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:22:59.434153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434116 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434170 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434219 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434255 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434328 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwb4\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-kube-api-access-hdwb4\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434367 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434483 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434635 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434705 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434705 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434776 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434832 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.434968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.434951 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.435284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.435259 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.436757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.436737 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.436835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.436760 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.458021 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.458002 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.458177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.458150 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwb4\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-kube-api-access-hdwb4\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.494411 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.494378 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:22:59.535898 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.535835 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-kube-api-access-b64zr\" (UniqueName: \"kubernetes.io/projected/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-kube-api-access-b64zr\") pod \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " Oct 30 00:22:59.536038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.535928 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-plugins-config-volume\") pod \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " Oct 30 00:22:59.536038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.535984 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-kube-api-access-sfxmw\" (UniqueName: \"kubernetes.io/projected/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-kube-api-access-sfxmw\") pod \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\" (UID: \"65c4887a-af7b-4fcc-a8f9-d5b3198da80a\") " Oct 30 00:22:59.536208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.536048 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-plugins-config-volume\") pod \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\" (UID: \"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074\") " Oct 30 00:22:59.536423 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.536386 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-plugins-config-volume" pod "64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" (UID: "64ebded0-f3c8-40b9-89d2-5c4e1f3e2074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:59.536423 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.536382 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-plugins-config-volume" pod "65c4887a-af7b-4fcc-a8f9-d5b3198da80a" (UID: "65c4887a-af7b-4fcc-a8f9-d5b3198da80a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:59.536969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.536947 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:59.537049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.536974 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:59.538120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.538094 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-kube-api-access-b64zr" pod "64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" (UID: "64ebded0-f3c8-40b9-89d2-5c4e1f3e2074"). InnerVolumeSpecName "kube-api-access-b64zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:59.538347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.538325 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-kube-api-access-sfxmw" pod "65c4887a-af7b-4fcc-a8f9-d5b3198da80a" (UID: "65c4887a-af7b-4fcc-a8f9-d5b3198da80a"). InnerVolumeSpecName "kube-api-access-sfxmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:59.638375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.638343 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64zr\" (UniqueName: \"kubernetes.io/projected/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074-kube-api-access-b64zr\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:59.638375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.638372 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxmw\" (UniqueName: \"kubernetes.io/projected/65c4887a-af7b-4fcc-a8f9-d5b3198da80a-kube-api-access-sfxmw\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:22:59.840553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.840523 4433 generic.go:356] "Generic (PLEG): container finished" podID="64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" containerID="44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa" exitCode=0 Oct 30 00:22:59.841073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.840568 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" Oct 30 00:22:59.841073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.840602 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" event={"ID":"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074","Type":"ContainerDied","Data":"44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa"} Oct 30 00:22:59.841073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.840661 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6" event={"ID":"64ebded0-f3c8-40b9-89d2-5c4e1f3e2074","Type":"ContainerDied","Data":"71a99fff076f6c0a86d800d84d5f3a9225f3fc7af76afa974c2ffc811c167337"} Oct 30 00:22:59.841073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.840691 4433 scope.go:122] "RemoveContainer" containerID="44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa" Oct 30 00:22:59.842180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.842160 4433 generic.go:356] "Generic (PLEG): container finished" podID="65c4887a-af7b-4fcc-a8f9-d5b3198da80a" containerID="84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6" exitCode=0 Oct 30 00:22:59.842292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.842206 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" Oct 30 00:22:59.842292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.842248 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" event={"ID":"65c4887a-af7b-4fcc-a8f9-d5b3198da80a","Type":"ContainerDied","Data":"84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6"} Oct 30 00:22:59.842382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.842303 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w" event={"ID":"65c4887a-af7b-4fcc-a8f9-d5b3198da80a","Type":"ContainerDied","Data":"7bcfd86fa9704681181b3491db8c1efac8b19668d60ad11966e494d4a8d5a875"} Oct 30 00:22:59.858161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.858144 4433 scope.go:122] "RemoveContainer" containerID="44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa" Oct 30 00:22:59.858449 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:22:59.858429 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa\": container with ID starting with 44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa not found: ID does not exist" containerID="44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa" Oct 30 00:22:59.858503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.858456 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa"} err="failed to get container status \"44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa\": rpc error: code = NotFound desc = could not find container \"44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa\": container with ID starting with 44113416bdbe789b99d1ac6095d9001e9e138d003e6b35d49a183acbdbf81afa not found: ID does not exist" Oct 30 00:22:59.858503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.858472 4433 scope.go:122] "RemoveContainer" containerID="84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6" Oct 30 00:22:59.868249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.868228 4433 scope.go:122] "RemoveContainer" containerID="84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6" Oct 30 00:22:59.868524 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:22:59.868501 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6\": container with ID starting with 84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6 not found: ID does not exist" containerID="84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6" Oct 30 00:22:59.868595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.868534 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6"} err="failed to get container status \"84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6\": rpc error: code = NotFound desc = could not find container \"84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6\": container with ID starting with 84edb1a41670ebfaa30b6d07a132dcef575059afb67a18331cf36bbc308dfbf6 not found: ID does not exist" Oct 30 00:22:59.897916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.897891 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w"] Oct 30 00:22:59.902119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.902093 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-84c8fcbb86-4pj8w"] Oct 30 00:22:59.937337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.937309 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6"] Oct 30 00:22:59.941793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:22:59.941763 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-69bb65ddd7-s8ts6"] Oct 30 00:23:00.374437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:00.374404 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:01.180623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:01.180592 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" path="/var/lib/kubelet/pods/64ebded0-f3c8-40b9-89d2-5c4e1f3e2074/volumes" Oct 30 00:23:01.181030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:01.181015 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c4887a-af7b-4fcc-a8f9-d5b3198da80a" path="/var/lib/kubelet/pods/65c4887a-af7b-4fcc-a8f9-d5b3198da80a/volumes" Oct 30 00:23:01.872903 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:01.872870 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:03.359724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:03.359687 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:04.411841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:04.411806 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:04.551468 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:04.551427 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:05.381324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:05.381283 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:06.737360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:06.736957 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr"] Oct 30 00:23:06.994469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:06.994382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:07.110088 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:23:07.110047 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:23:07.156666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:07.155907 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr"] Oct 30 00:23:07.166310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:07.166266 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:23:07.166557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:07.166527 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:23:07.166817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:07.166793 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:23:07.922148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:07.922107 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" event={"ID":"73c6a0e5-d146-4014-9851-0cb3734cc5f0","Type":"ContainerStarted","Data":"03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b"} Oct 30 00:23:07.922584 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:07.922174 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" event={"ID":"73c6a0e5-d146-4014-9851-0cb3734cc5f0","Type":"ContainerStarted","Data":"89d8667bfe11bda78fb0beae3e112182f23468dffe91fcc8ae2f4f74d8f163d5"} Oct 30 00:23:07.981745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:07.981696 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podStartSLOduration=8.9816822 podStartE2EDuration="8.9816822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:07.980388905 +0000 UTC m=+1096.914457460" watchObservedRunningTime="2026-10-30 00:23:07.9816822 +0000 UTC m=+1096.915750735" Oct 30 00:23:08.495478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:08.495439 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:23:08.497000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:08.496973 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 00:23:08.497135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:08.497026 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 00:23:08.623961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:08.623917 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:09.495255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:09.495217 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 00:23:09.495792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:09.495293 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 00:23:09.878420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:09.878370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:10.006266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:10.006231 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:10.495479 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:10.495433 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 00:23:10.496079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:10.495507 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 00:23:10.985048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:10.985007 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:11.495979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:11.495942 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 00:23:11.496472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:11.496002 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 00:23:12.494995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:12.494959 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 00:23:12.495182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:12.495019 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 00:23:12.606846 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:12.606754 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:13.495091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:13.495054 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 00:23:13.495258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:13.495117 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 00:23:14.184291 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:14.184259 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:14.494998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:14.494914 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:23:14.500238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:14.500210 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:23:14.998135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:14.998103 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:23:15.644587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:15.644544 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:15.751262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:15.751229 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:15.856478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:15.856438 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:16.541758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:16.541708 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:16.650544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:16.650503 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:17.230180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:17.230147 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:18.252527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:18.252493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:18.366622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:18.366581 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:18.786253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:18.783088 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:18.978270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:18.978235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:20.293479 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:20.293444 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:21.874837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:21.874800 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:23.317601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:23.317540 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:24.799716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:24.799679 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:24.969169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:24.969129 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:26.324880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:26.324828 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:26.441733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:26.441695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:26.851438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:26.851382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:26.966149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:26.966116 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:27.094598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.082621 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:27.565676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.565640 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4"] Oct 30 00:23:27.566687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.566664 4433 cpu_manager.go:414] "containerMap: removing container" podUID="65c4887a-af7b-4fcc-a8f9-d5b3198da80a" containerName="payload-processing" Oct 30 00:23:27.566687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.566687 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c4887a-af7b-4fcc-a8f9-d5b3198da80a" containerName="payload-processing" Oct 30 00:23:27.566872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.566720 4433 cpu_manager.go:414] "containerMap: removing container" podUID="64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" containerName="payload-pre-processing" Oct 30 00:23:27.566872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.566728 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" containerName="payload-pre-processing" Oct 30 00:23:27.566984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.566933 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ebded0-f3c8-40b9-89d2-5c4e1f3e2074" containerName="payload-pre-processing" Oct 30 00:23:27.566984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.566964 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c4887a-af7b-4fcc-a8f9-d5b3198da80a" containerName="payload-processing" Oct 30 00:23:27.571183 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.571161 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.592752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.592704 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4"] Oct 30 00:23:27.599752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.599724 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6a55e1-openshift-default-dockercfg-fftxr\"" Oct 30 00:23:27.663700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.663665 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:27.727193 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.712891 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-certs\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.727470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.727430 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-podinfo\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.727588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.727516 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-credential-socket\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.727659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.727610 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-socket\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.727748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.727732 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-envoy\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.727938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.727917 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-token\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.728840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.728802 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvh7\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-kube-api-access-gcvh7\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.730099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.729926 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-data\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.730316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.730292 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee9f6b-478a-42c9-9bb2-774948b06a17-istiod-ca-cert\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.832157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.832046 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-data\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.833186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.832167 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee9f6b-478a-42c9-9bb2-774948b06a17-istiod-ca-cert\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.833186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.832292 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-certs\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.833186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.832326 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-podinfo\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.833186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.832384 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-credential-socket\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.832455 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-socket\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.834928 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-envoy\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.835015 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-credential-socket\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.835110 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-socket\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.835259 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee9f6b-478a-42c9-9bb2-774948b06a17-istiod-ca-cert\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.835171 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-token\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.835336 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvh7\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-kube-api-access-gcvh7\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.835416 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-data\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.835824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.835470 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-certs\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.837648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.837623 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-envoy\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.837964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.837946 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-podinfo\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.868306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.868269 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-token\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.870193 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.870168 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvh7\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-kube-api-access-gcvh7\") pod \"e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.891927 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.891568 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:27.896409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:27.896382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:28.253091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:28.253058 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4"] Oct 30 00:23:28.254380 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:23:28.254313 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ee9f6b_478a_42c9_9bb2_774948b06a17.slice/crio-9e43eff8bb010ad89a69ab0cfc197298137763ce87ef43fb45f8b4bbd9ff74c4 WatchSource:0}: Error finding container 9e43eff8bb010ad89a69ab0cfc197298137763ce87ef43fb45f8b4bbd9ff74c4: Status 404 returned error can't find the container with id 9e43eff8bb010ad89a69ab0cfc197298137763ce87ef43fb45f8b4bbd9ff74c4 Oct 30 00:23:28.257817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:28.257779 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:23:28.257985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:28.257867 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:23:28.257985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:28.257945 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:23:28.483678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:28.483634 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:29.181975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:29.181937 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" event={"ID":"75ee9f6b-478a-42c9-9bb2-774948b06a17","Type":"ContainerStarted","Data":"9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c"} Oct 30 00:23:29.182372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:29.181991 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" event={"ID":"75ee9f6b-478a-42c9-9bb2-774948b06a17","Type":"ContainerStarted","Data":"9e43eff8bb010ad89a69ab0cfc197298137763ce87ef43fb45f8b4bbd9ff74c4"} Oct 30 00:23:29.239779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:29.239723 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podStartSLOduration=2.239708803 podStartE2EDuration="2.239708803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:29.237493581 +0000 UTC m=+1118.171562115" watchObservedRunningTime="2026-10-30 00:23:29.239708803 +0000 UTC m=+1118.173777398" Oct 30 00:23:29.895707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:29.895647 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:29.908831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:29.908797 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:30.022500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:30.022461 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:30.186326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:30.186229 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:30.187248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:30.187228 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:23:31.651477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:31.651447 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:31.778789 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:31.778754 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:32.236383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:32.231480 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:32.344123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:32.344089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:33.330277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:33.330239 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:33.436714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:33.436679 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:34.190543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:34.190510 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:35.091486 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.091376 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:35.204508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.204467 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:35.382505 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.382411 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7"] Oct 30 00:23:35.402421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.400536 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:35.428155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.428116 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7"] Oct 30 00:23:35.429430 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.429232 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96027e\"" Oct 30 00:23:35.461600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.460949 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96027e-dockercfg-mzpcl\"" Oct 30 00:23:35.564764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.547435 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:35.567711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.560484 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qch4\" (UniqueName: \"kubernetes.io/projected/a12100cb-774e-4e35-b9e9-a192fa30537b-kube-api-access-5qch4\") pod \"payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:35.568164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.568139 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12100cb-774e-4e35-b9e9-a192fa30537b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:35.670372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.670289 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qch4\" (UniqueName: \"kubernetes.io/projected/a12100cb-774e-4e35-b9e9-a192fa30537b-kube-api-access-5qch4\") pod \"payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:35.670516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.670442 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12100cb-774e-4e35-b9e9-a192fa30537b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:35.671278 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.671255 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12100cb-774e-4e35-b9e9-a192fa30537b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:35.727596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.727341 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qch4\" (UniqueName: \"kubernetes.io/projected/a12100cb-774e-4e35-b9e9-a192fa30537b-kube-api-access-5qch4\") pod \"payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:35.935150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:35.933832 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:36.041594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:36.040601 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:36.150256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:36.150081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:36.255492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:36.255395 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:36.370301 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:36.370268 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7"] Oct 30 00:23:36.370424 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:23:36.370317 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12100cb_774e_4e35_b9e9_a192fa30537b.slice/crio-6b6b2ed2c747cf3cd6a5e0cee30b9d2d6aa0e183298fe68ee52f550fa4713d20 WatchSource:0}: Error finding container 6b6b2ed2c747cf3cd6a5e0cee30b9d2d6aa0e183298fe68ee52f550fa4713d20: Status 404 returned error can't find the container with id 6b6b2ed2c747cf3cd6a5e0cee30b9d2d6aa0e183298fe68ee52f550fa4713d20 Oct 30 00:23:37.101422 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:23:37.101385 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:23:37.245896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:37.245863 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" event={"ID":"a12100cb-774e-4e35-b9e9-a192fa30537b","Type":"ContainerStarted","Data":"935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02"} Oct 30 00:23:37.246309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:37.245929 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:37.246309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:37.245948 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" event={"ID":"a12100cb-774e-4e35-b9e9-a192fa30537b","Type":"ContainerStarted","Data":"6b6b2ed2c747cf3cd6a5e0cee30b9d2d6aa0e183298fe68ee52f550fa4713d20"} Oct 30 00:23:37.288797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:37.288746 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" podStartSLOduration=2.288732904 podStartE2EDuration="2.288732904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:37.286881941 +0000 UTC m=+1126.220950477" watchObservedRunningTime="2026-10-30 00:23:37.288732904 +0000 UTC m=+1126.222801440" Oct 30 00:23:37.697670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:37.697629 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:37.892157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:37.892121 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:39.468974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:39.468934 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:40.725550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:40.725511 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:40.968545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:40.968492 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:42.540910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:42.540874 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:44.052892 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:44.052853 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:45.580005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:45.579967 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:47.305075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:47.305024 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:47.964968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:47.964932 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:48.070485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:48.070447 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:48.258430 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:48.258348 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:23:48.756390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:48.756355 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:48.888249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:48.888222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:49.454589 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:49.454541 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:49.998420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:49.998382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:50.109336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:50.109305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:50.213690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:50.213657 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:50.624406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:50.624367 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:51.704642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:51.704563 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:53.114191 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:53.114155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:53.232780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:53.232750 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:53.344766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:53.344726 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:54.495288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.495241 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp"] Oct 30 00:23:54.499698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.499674 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.528506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.528470 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp"] Oct 30 00:23:54.529918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.529895 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-96027e-serving-cert\"" Oct 30 00:23:54.570175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.570137 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-metrics-certs\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.570393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.570201 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d30c41b-626d-4e4b-8d0b-5211b589275d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.570393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.570327 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bnl\" (UniqueName: \"kubernetes.io/projected/2d30c41b-626d-4e4b-8d0b-5211b589275d-kube-api-access-q5bnl\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.570524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.570414 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.580495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.580464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:54.672495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.672464 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-metrics-certs\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.672685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.672508 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d30c41b-626d-4e4b-8d0b-5211b589275d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.672685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.672648 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bnl\" (UniqueName: \"kubernetes.io/projected/2d30c41b-626d-4e4b-8d0b-5211b589275d-kube-api-access-q5bnl\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.672801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.672729 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.673198 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.673175 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d30c41b-626d-4e4b-8d0b-5211b589275d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.674928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.674910 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-metrics-certs\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.675009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.674963 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.696213 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.696184 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bnl\" (UniqueName: \"kubernetes.io/projected/2d30c41b-626d-4e4b-8d0b-5211b589275d-kube-api-access-q5bnl\") pod \"maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:54.712944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.708736 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:54.814553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:54.814520 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:55.143659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:55.143625 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp"] Oct 30 00:23:55.170206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:55.170167 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:55.284308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:55.283869 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:55.420226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:55.420123 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" event={"ID":"2d30c41b-626d-4e4b-8d0b-5211b589275d","Type":"ContainerStarted","Data":"c40257571dc8b98a9428eb08adcfd8e8c300f5ef8b20c0404ff6a190b15088d5"} Oct 30 00:23:56.301757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:56.301717 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:56.431401 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:56.431157 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" event={"ID":"2d30c41b-626d-4e4b-8d0b-5211b589275d","Type":"ContainerStarted","Data":"a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350"} Oct 30 00:23:56.431401 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:56.431250 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:23:56.478152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:56.478092 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" podStartSLOduration=2.080074071 podStartE2EDuration="2.478076282s" totalImagesPullingTime="398.002211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:56.47774936 +0000 UTC m=+1145.411817897" watchObservedRunningTime="2026-10-30 00:23:56.478076282 +0000 UTC m=+1145.412144816" Oct 30 00:23:57.622304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:57.622269 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:57.740590 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:57.740541 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:57.991499 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:57.991422 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j"] Oct 30 00:23:57.991760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:57.991732 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" containerID="cri-o://266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de" gracePeriod=30 Oct 30 00:23:58.021925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:58.020335 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:58.385408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:58.385376 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:23:59.651902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:23:59.651865 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.034609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.034499 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.147968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.147928 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.258141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.258104 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.377190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.377094 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.487628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.484919 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.667705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.665936 4433 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:24:00.667705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.666006 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:24:00.714047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.714010 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.820864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.820828 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:00.933011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:00.932901 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:01.925481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:01.925447 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:02.445017 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:02.444988 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:24:02.446259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:02.446232 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:02.590806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:02.590772 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:02.713316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:02.713213 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:03.195134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.195102 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:03.315113 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.315086 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.473510 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-token\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-token\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.473738 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istiod-ca-cert\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.473920 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-data\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.473993 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-kube-api-access-l5c8n\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-kube-api-access-l5c8n\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.474033 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-socket\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.474061 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-podinfo\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.474088 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-credential-socket\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.474138 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-envoy\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.474525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.474199 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-certs\") pod \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\" (UID: \"a11a8c90-f95d-4aa1-a5ea-067a280a395a\") " Oct 30 00:24:03.475332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.474739 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-socket" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:03.475332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.475016 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-certs" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:03.475539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.475512 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.475680 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.475544 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.475745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.475688 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-data" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:03.476512 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.476042 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istiod-ca-cert" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:24:03.476512 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.476131 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-credential-socket" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:03.476986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.476961 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-token" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:03.477702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.477673 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-podinfo" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:24:03.478062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.478042 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-kube-api-access-l5c8n" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "kube-api-access-l5c8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:03.478152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.478118 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-envoy" pod "a11a8c90-f95d-4aa1-a5ea-067a280a395a" (UID: "a11a8c90-f95d-4aa1-a5ea-067a280a395a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:03.542939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.542829 4433 generic.go:356] "Generic (PLEG): container finished" podID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerID="266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de" exitCode=0 Oct 30 00:24:03.543450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.543420 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" event={"ID":"a11a8c90-f95d-4aa1-a5ea-067a280a395a","Type":"ContainerDied","Data":"266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de"} Oct 30 00:24:03.543729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.543583 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" Oct 30 00:24:03.544813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.543720 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j" event={"ID":"a11a8c90-f95d-4aa1-a5ea-067a280a395a","Type":"ContainerDied","Data":"d0953bb25658aa2046678a3d2f6dea40b55a7dee8c256faf5d1d59466b0b59f9"} Oct 30 00:24:03.544922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.543739 4433 scope.go:122] "RemoveContainer" containerID="266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de" Oct 30 00:24:03.558988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.558959 4433 scope.go:122] "RemoveContainer" containerID="266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de" Oct 30 00:24:03.559323 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:24:03.559298 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de\": container with ID starting with 266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de not found: ID does not exist" containerID="266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de" Oct 30 00:24:03.559391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.559336 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de"} err="failed to get container status \"266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de\": rpc error: code = NotFound desc = could not find container \"266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de\": container with ID starting with 266a1fb550e629186b1761a03d6d9eefd63b7c101f054ed60301cf2c788af7de not found: ID does not exist" Oct 30 00:24:03.577472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.577446 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.577472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.577474 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5c8n\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-kube-api-access-l5c8n\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.577682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.577487 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.577682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.577501 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.577682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.577512 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.577682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.577524 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.577682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.577535 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11a8c90-f95d-4aa1-a5ea-067a280a395a-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:03.602478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.602448 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j"] Oct 30 00:24:03.607756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:03.607731 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-gxb7j"] Oct 30 00:24:04.320745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:04.320706 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:05.181961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:05.181928 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" path="/var/lib/kubelet/pods/a11a8c90-f95d-4aa1-a5ea-067a280a395a/volumes" Oct 30 00:24:05.799630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:05.799596 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:07.100464 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:24:07.100421 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:24:07.301331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:07.301293 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:07.475778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:07.475688 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:07.595037 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:07.595001 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:07.861129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:07.861097 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:09.065820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:09.065782 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:09.676507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:09.676471 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:09.820717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:09.820678 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:10.580253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:10.580214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:12.081886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:12.081849 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:12.638509 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:12.638475 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:12.750235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:12.750205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:13.581284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:13.581253 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:13.631053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:13.631010 4433 generic.go:356] "Generic (PLEG): container finished" podID="6714f311-dfc3-4727-b9b8-018915d05567" containerID="360beb2bfd320f233e7e11c260abd1b83e6a625fbe6937d9139843409f26a54a" exitCode=1 Oct 30 00:24:13.631834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:13.631801 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" event={"ID":"6714f311-dfc3-4727-b9b8-018915d05567","Type":"ContainerDied","Data":"360beb2bfd320f233e7e11c260abd1b83e6a625fbe6937d9139843409f26a54a"} Oct 30 00:24:13.632759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:13.632392 4433 scope.go:122] "RemoveContainer" containerID="360beb2bfd320f233e7e11c260abd1b83e6a625fbe6937d9139843409f26a54a" Oct 30 00:24:14.643090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:14.643051 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" event={"ID":"6714f311-dfc3-4727-b9b8-018915d05567","Type":"ContainerStarted","Data":"7d02715fc079f3116ba154b953932774895140ca4fdbecc8d136cef53eb0c8ca"} Oct 30 00:24:14.643470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:14.643253 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:24:15.324916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:15.324878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:16.713869 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:16.713832 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:18.078501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:18.078462 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:19.719931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:19.719895 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:20.041135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:20.041036 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:21.506835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:21.506538 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:22.933217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:22.933172 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:23.045901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:23.045868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:23.151416 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:23.151382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:23.660598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:23.651232 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:23.846988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:23.846100 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7"] Oct 30 00:24:24.016702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:24.007108 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:24.124637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:24.124566 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:24.832674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:24.831135 4433 generic.go:356] "Generic (PLEG): container finished" podID="6714f311-dfc3-4727-b9b8-018915d05567" containerID="7d02715fc079f3116ba154b953932774895140ca4fdbecc8d136cef53eb0c8ca" exitCode=1 Oct 30 00:24:24.832674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:24.831305 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" event={"ID":"6714f311-dfc3-4727-b9b8-018915d05567","Type":"ContainerDied","Data":"7d02715fc079f3116ba154b953932774895140ca4fdbecc8d136cef53eb0c8ca"} Oct 30 00:24:24.832674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:24.831347 4433 scope.go:122] "RemoveContainer" containerID="360beb2bfd320f233e7e11c260abd1b83e6a625fbe6937d9139843409f26a54a" Oct 30 00:24:24.832674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:24.831910 4433 scope.go:122] "RemoveContainer" containerID="7d02715fc079f3116ba154b953932774895140ca4fdbecc8d136cef53eb0c8ca" Oct 30 00:24:24.832674 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:24:24.832406 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=jobset-controller-manager-5ffdccb548-wsvmp_openshift-jobset-operator(6714f311-dfc3-4727-b9b8-018915d05567)\"" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" podUID="6714f311-dfc3-4727-b9b8-018915d05567" Oct 30 00:24:25.645453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:25.645392 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:27.277215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:27.277158 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:28.799252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:28.799220 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:28.846771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:28.846729 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" podUID="a12100cb-774e-4e35-b9e9-a192fa30537b" containerName="payload-processing" containerID="cri-o://935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02" gracePeriod=25 Oct 30 00:24:29.162107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.162079 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:24:29.206148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.206121 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12100cb-774e-4e35-b9e9-a192fa30537b-kube-api-access-5qch4\" (UniqueName: \"kubernetes.io/projected/a12100cb-774e-4e35-b9e9-a192fa30537b-kube-api-access-5qch4\") pod \"a12100cb-774e-4e35-b9e9-a192fa30537b\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " Oct 30 00:24:29.206369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.206320 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12100cb-774e-4e35-b9e9-a192fa30537b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12100cb-774e-4e35-b9e9-a192fa30537b-plugins-config-volume\") pod \"a12100cb-774e-4e35-b9e9-a192fa30537b\" (UID: \"a12100cb-774e-4e35-b9e9-a192fa30537b\") " Oct 30 00:24:29.207457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.207426 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12100cb-774e-4e35-b9e9-a192fa30537b-plugins-config-volume" pod "a12100cb-774e-4e35-b9e9-a192fa30537b" (UID: "a12100cb-774e-4e35-b9e9-a192fa30537b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:24:29.215126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.210048 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12100cb-774e-4e35-b9e9-a192fa30537b-kube-api-access-5qch4" pod "a12100cb-774e-4e35-b9e9-a192fa30537b" (UID: "a12100cb-774e-4e35-b9e9-a192fa30537b"). InnerVolumeSpecName "kube-api-access-5qch4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:29.308745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.308708 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qch4\" (UniqueName: \"kubernetes.io/projected/a12100cb-774e-4e35-b9e9-a192fa30537b-kube-api-access-5qch4\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:29.308745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.308738 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12100cb-774e-4e35-b9e9-a192fa30537b-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:24:29.897591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.897529 4433 generic.go:356] "Generic (PLEG): container finished" podID="a12100cb-774e-4e35-b9e9-a192fa30537b" containerID="935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02" exitCode=0 Oct 30 00:24:29.898139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.897601 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" event={"ID":"a12100cb-774e-4e35-b9e9-a192fa30537b","Type":"ContainerDied","Data":"935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02"} Oct 30 00:24:29.898139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.897610 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" Oct 30 00:24:29.898139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.897704 4433 scope.go:122] "RemoveContainer" containerID="935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02" Oct 30 00:24:29.898139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.897692 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7" event={"ID":"a12100cb-774e-4e35-b9e9-a192fa30537b","Type":"ContainerDied","Data":"6b6b2ed2c747cf3cd6a5e0cee30b9d2d6aa0e183298fe68ee52f550fa4713d20"} Oct 30 00:24:29.916516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.916493 4433 scope.go:122] "RemoveContainer" containerID="935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02" Oct 30 00:24:29.916859 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:24:29.916831 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02\": container with ID starting with 935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02 not found: ID does not exist" containerID="935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02" Oct 30 00:24:29.916985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.916865 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02"} err="failed to get container status \"935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02\": rpc error: code = NotFound desc = could not find container \"935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02\": container with ID starting with 935a60a71ef1c5a03f06013797858062bdb5c7ef2683532c9a5b85eca2cc8c02 not found: ID does not exist" Oct 30 00:24:29.956073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.956041 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7"] Oct 30 00:24:29.967123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:29.967090 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-5bb58c4bdd-cdpv7"] Oct 30 00:24:30.174185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.174089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:30.698049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.696518 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:40038->10.134.0.105:15021: read: connection reset by peer" start-of-body= Oct 30 00:24:30.698049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.696643 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:40038->10.134.0.105:15021: read: connection reset by peer" Oct 30 00:24:30.906723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.906638 4433 generic.go:356] "Generic (PLEG): container finished" podID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerID="9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c" exitCode=0 Oct 30 00:24:30.907189 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.906719 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" event={"ID":"75ee9f6b-478a-42c9-9bb2-774948b06a17","Type":"ContainerDied","Data":"9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c"} Oct 30 00:24:30.907283 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.907266 4433 scope.go:122] "RemoveContainer" containerID="9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c" Oct 30 00:24:30.909114 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.909085 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:24:30.909189 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.909170 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:24:30.909243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:30.909203 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:24:31.185691 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:31.185653 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12100cb-774e-4e35-b9e9-a192fa30537b" path="/var/lib/kubelet/pods/a12100cb-774e-4e35-b9e9-a192fa30537b/volumes" Oct 30 00:24:31.521787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:31.521749 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:31.917510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:31.917472 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" event={"ID":"75ee9f6b-478a-42c9-9bb2-774948b06a17","Type":"ContainerStarted","Data":"599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3"} Oct 30 00:24:32.895744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:32.895712 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:24:32.895924 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:32.895891 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 00:24:32.895966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:32.895934 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 00:24:32.989235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:32.989199 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:33.895456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:33.895420 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 00:24:33.895675 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:33.895478 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 00:24:34.621814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:34.621775 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:34.895266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:34.895169 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 00:24:34.895266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:34.895233 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 00:24:35.895826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:35.895788 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 00:24:35.896269 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:35.895847 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 00:24:36.096883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:36.096848 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:36.895087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:36.895047 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 00:24:36.895266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:36.895101 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 00:24:37.108607 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:24:37.106851 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:24:37.431232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.431193 4433 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:24:37.431745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.431726 4433 scope.go:122] "RemoveContainer" containerID="7d02715fc079f3116ba154b953932774895140ca4fdbecc8d136cef53eb0c8ca" Oct 30 00:24:37.576096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.576059 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:37.899713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.899681 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:24:37.900089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.900069 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:24:37.900804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.900783 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:24:37.948218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.948190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:37.987984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.987949 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" event={"ID":"6714f311-dfc3-4727-b9b8-018915d05567","Type":"ContainerStarted","Data":"442c40adab34653043252f0589219b335f05a86a940dcbf2ba6e6ec79297c6eb"} Oct 30 00:24:37.988208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:37.988194 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:24:39.441620 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:39.441555 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:40.792050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:40.792014 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:42.308383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:42.308336 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:43.298532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:43.298500 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:43.425105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:43.425067 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:44.322612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:44.322566 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:45.343032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.342986 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:45.468356 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.468304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:45.773171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.773084 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f"] Oct 30 00:24:45.777774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.777737 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a12100cb-774e-4e35-b9e9-a192fa30537b" containerName="payload-processing" Oct 30 00:24:45.777969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.777955 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12100cb-774e-4e35-b9e9-a192fa30537b" containerName="payload-processing" Oct 30 00:24:45.778140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.778130 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" Oct 30 00:24:45.778207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.778199 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" Oct 30 00:24:45.778595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.778565 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12100cb-774e-4e35-b9e9-a192fa30537b" containerName="payload-processing" Oct 30 00:24:45.778758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.778744 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11a8c90-f95d-4aa1-a5ea-067a280a395a" containerName="istio-proxy" Oct 30 00:24:45.786452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.786422 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f"] Oct 30 00:24:45.786730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.786626 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:45.841937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.839711 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96027e\"" Oct 30 00:24:45.841937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.839845 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96027e-dockercfg-m7vjn\"" Oct 30 00:24:45.870108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.870067 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbjhb\" (UniqueName: \"kubernetes.io/projected/1cd4cf6e-0a68-44d4-959e-087bd8b443af-kube-api-access-xbjhb\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:45.870834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.870473 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd4cf6e-0a68-44d4-959e-087bd8b443af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:45.972806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.972758 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd4cf6e-0a68-44d4-959e-087bd8b443af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:45.973025 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.972950 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbjhb\" (UniqueName: \"kubernetes.io/projected/1cd4cf6e-0a68-44d4-959e-087bd8b443af-kube-api-access-xbjhb\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:45.973550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.973520 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd4cf6e-0a68-44d4-959e-087bd8b443af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:45.997111 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:45.997075 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbjhb\" (UniqueName: \"kubernetes.io/projected/1cd4cf6e-0a68-44d4-959e-087bd8b443af-kube-api-access-xbjhb\") pod \"payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:46.146927 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:46.146883 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:46.304033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:46.303999 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:46.465976 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:24:46.464939 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd4cf6e_0a68_44d4_959e_087bd8b443af.slice/crio-327d6b50fd2e70fc40bf98ec44971e8e4643392ed886c71ebb831a9f2cf04912 WatchSource:0}: Error finding container 327d6b50fd2e70fc40bf98ec44971e8e4643392ed886c71ebb831a9f2cf04912: Status 404 returned error can't find the container with id 327d6b50fd2e70fc40bf98ec44971e8e4643392ed886c71ebb831a9f2cf04912 Oct 30 00:24:46.469074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:46.469024 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f"] Oct 30 00:24:47.081125 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:47.081083 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" event={"ID":"1cd4cf6e-0a68-44d4-959e-087bd8b443af","Type":"ContainerStarted","Data":"327d6b50fd2e70fc40bf98ec44971e8e4643392ed886c71ebb831a9f2cf04912"} Oct 30 00:24:48.095239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:48.095201 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" event={"ID":"1cd4cf6e-0a68-44d4-959e-087bd8b443af","Type":"ContainerStarted","Data":"11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7"} Oct 30 00:24:48.095765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:48.095322 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:48.141729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:48.141664 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" podStartSLOduration=1.978390021 podStartE2EDuration="3.141645775s" totalImagesPullingTime="1.163255754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:48.136408721 +0000 UTC m=+1197.070477308" watchObservedRunningTime="2026-10-30 00:24:48.141645775 +0000 UTC m=+1197.075714309" Oct 30 00:24:48.268886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:48.268843 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:49.000594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:49.000549 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5ffdccb548-wsvmp" Oct 30 00:24:49.676514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:49.676469 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:49.942160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:49.942047 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:50.386457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.386424 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj"] Oct 30 00:24:50.392456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.392428 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.410618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.408527 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3fb3ad-openshift-default-dockercfg-mwsjt\"" Oct 30 00:24:50.429175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.429137 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj"] Oct 30 00:24:50.453019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.452984 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.576746 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-envoy\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.576847 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-data\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.577013 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-certs\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.577050 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-podinfo\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.577089 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.577322 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8jg\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-kube-api-access-9h8jg\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.577387 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-token\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.577416 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istiod-ca-cert\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.579107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.577453 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-credential-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.687969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.687867 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-token\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.688607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.688561 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istiod-ca-cert\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.688762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.688745 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-credential-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.688934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.688918 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-envoy\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.689217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.689197 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-data\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.689643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.689618 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-certs\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.689797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.689781 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-podinfo\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.689936 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.689904 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.690219 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.690197 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8jg\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-kube-api-access-9h8jg\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.691715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.691692 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istiod-ca-cert\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.692547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.692525 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-credential-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.693830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.693145 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-certs\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.694205 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.694186 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.705234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.694545 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-data\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.708659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.708153 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-podinfo\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.732597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.721438 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-envoy\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.742117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.741322 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-token\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.765598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.761987 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8jg\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-kube-api-access-9h8jg\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:50.897287 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:50.897256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:51.009406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.009319 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:51.062600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.062138 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:51.170926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.170641 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:51.320009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.319952 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:51.381060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.381027 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj"] Oct 30 00:24:51.383797 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:24:51.383769 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebc67f8_bd38_44fd_ab6d_42b50bd609fc.slice/crio-1f5be3e8aaf50ee43f19e23854f90dede68108fc53f493f34f8cafdc47495fa8 WatchSource:0}: Error finding container 1f5be3e8aaf50ee43f19e23854f90dede68108fc53f493f34f8cafdc47495fa8: Status 404 returned error can't find the container with id 1f5be3e8aaf50ee43f19e23854f90dede68108fc53f493f34f8cafdc47495fa8 Oct 30 00:24:51.387414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.387373 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:24:51.387506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.387456 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:24:51.387506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:51.387488 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:24:52.150880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:52.150834 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" event={"ID":"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc","Type":"ContainerStarted","Data":"0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40"} Oct 30 00:24:52.151344 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:52.150942 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" event={"ID":"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc","Type":"ContainerStarted","Data":"1f5be3e8aaf50ee43f19e23854f90dede68108fc53f493f34f8cafdc47495fa8"} Oct 30 00:24:52.225530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:52.225472 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podStartSLOduration=2.225457997 podStartE2EDuration="2.225457997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:52.210197006 +0000 UTC m=+1201.144265541" watchObservedRunningTime="2026-10-30 00:24:52.225457997 +0000 UTC m=+1201.159526532" Oct 30 00:24:52.715266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:52.715222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:52.835159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:52.835122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:53.077229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.075174 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:53.077229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.075243 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w"] Oct 30 00:24:53.107848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.103280 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w"] Oct 30 00:24:53.107848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.103419 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.218678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.204362 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f8397b-dockercfg-2sx76\"" Oct 30 00:24:53.218678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.211338 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f8397b\"" Oct 30 00:24:53.218678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.211489 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.218678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.211735 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmb9\" (UniqueName: \"kubernetes.io/projected/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-kube-api-access-kdmb9\") pod \"payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.230709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.230660 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:53.250806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.250776 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:53.250972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.250844 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:53.250972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.250944 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:24:53.313632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.313548 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.313824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.313731 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmb9\" (UniqueName: \"kubernetes.io/projected/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-kube-api-access-kdmb9\") pod \"payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.314407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.314385 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.337919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.337840 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmb9\" (UniqueName: \"kubernetes.io/projected/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-kube-api-access-kdmb9\") pod \"payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.560962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.560922 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:53.909180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:53.909146 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w"] Oct 30 00:24:53.909383 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:24:53.909356 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba9c019_43cf_4e31_a9ad_1a887c2be5ff.slice/crio-903c321387261500deb0baeda67b1618dd8c1423bdf1cadbc0e2ed7a4553306a WatchSource:0}: Error finding container 903c321387261500deb0baeda67b1618dd8c1423bdf1cadbc0e2ed7a4553306a: Status 404 returned error can't find the container with id 903c321387261500deb0baeda67b1618dd8c1423bdf1cadbc0e2ed7a4553306a Oct 30 00:24:54.257623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:54.257510 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" event={"ID":"fba9c019-43cf-4e31-a9ad-1a887c2be5ff","Type":"ContainerStarted","Data":"a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9"} Oct 30 00:24:54.257623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:54.257608 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" event={"ID":"fba9c019-43cf-4e31-a9ad-1a887c2be5ff","Type":"ContainerStarted","Data":"903c321387261500deb0baeda67b1618dd8c1423bdf1cadbc0e2ed7a4553306a"} Oct 30 00:24:54.258108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:54.257836 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:24:54.302156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:54.302098 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" podStartSLOduration=2.302085136 podStartE2EDuration="2.302085136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:54.298363043 +0000 UTC m=+1203.232431597" watchObservedRunningTime="2026-10-30 00:24:54.302085136 +0000 UTC m=+1203.236153670" Oct 30 00:24:54.534818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:54.534727 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:54.651920 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:54.651888 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:54.795964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:54.795884 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:55.359900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:55.359864 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:24:55.369815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:55.369789 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:24:56.207119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.207081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:56.314482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.314451 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:56.435431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.435394 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:56.477838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.477733 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt"] Oct 30 00:24:56.500916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.500879 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:56.528400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.528360 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt"] Oct 30 00:24:56.552341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.550088 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fb3ad-dockercfg-r5758\"" Oct 30 00:24:56.552341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.550203 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fb3ad\"" Oct 30 00:24:56.614211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.590250 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdzm\" (UniqueName: \"kubernetes.io/projected/b2c47144-8118-490f-82ec-e2ee06a52b7c-kube-api-access-gwdzm\") pod \"payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:56.614211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.590429 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47144-8118-490f-82ec-e2ee06a52b7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:56.713701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.711523 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdzm\" (UniqueName: \"kubernetes.io/projected/b2c47144-8118-490f-82ec-e2ee06a52b7c-kube-api-access-gwdzm\") pod \"payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:56.714233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.714197 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47144-8118-490f-82ec-e2ee06a52b7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:56.715847 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.715823 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47144-8118-490f-82ec-e2ee06a52b7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:56.743212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:56.743094 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdzm\" (UniqueName: \"kubernetes.io/projected/b2c47144-8118-490f-82ec-e2ee06a52b7c-kube-api-access-gwdzm\") pod \"payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:57.040589 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:57.040463 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:57.362709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:57.362676 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt"] Oct 30 00:24:57.363128 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:24:57.363099 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c47144_8118_490f_82ec_e2ee06a52b7c.slice/crio-a44b5004fa680e6d6e2205d513bd2acc544aee5c2e8c64f01f4f362befc1c37e WatchSource:0}: Error finding container a44b5004fa680e6d6e2205d513bd2acc544aee5c2e8c64f01f4f362befc1c37e: Status 404 returned error can't find the container with id a44b5004fa680e6d6e2205d513bd2acc544aee5c2e8c64f01f4f362befc1c37e Oct 30 00:24:57.965803 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:57.965769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:24:58.303169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:58.303068 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" event={"ID":"b2c47144-8118-490f-82ec-e2ee06a52b7c","Type":"ContainerStarted","Data":"f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e"} Oct 30 00:24:58.303169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:58.303169 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:24:58.303405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:58.303188 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" event={"ID":"b2c47144-8118-490f-82ec-e2ee06a52b7c","Type":"ContainerStarted","Data":"a44b5004fa680e6d6e2205d513bd2acc544aee5c2e8c64f01f4f362befc1c37e"} Oct 30 00:24:58.348662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:58.348536 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" podStartSLOduration=2.348517254 podStartE2EDuration="2.348517254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:58.347297198 +0000 UTC m=+1207.281365744" watchObservedRunningTime="2026-10-30 00:24:58.348517254 +0000 UTC m=+1207.282585789" Oct 30 00:24:59.106647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:59.106613 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:24:59.371448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:24:59.371363 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:00.202324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.202282 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57"] Oct 30 00:25:00.202879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.202554 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" containerID="cri-o://16b2cfd916aca20744e2346a53c41388dc2242303409953a34489a640e620544" gracePeriod=30 Oct 30 00:25:00.215844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.215764 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj"] Oct 30 00:25:00.216046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.216020 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" containerID="cri-o://9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741" gracePeriod=30 Oct 30 00:25:00.220764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.220741 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg"] Oct 30 00:25:00.220929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.220910 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" containerID="cri-o://dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f" gracePeriod=30 Oct 30 00:25:00.815295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.815258 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb"] Oct 30 00:25:00.848206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.848163 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l"] Oct 30 00:25:00.860375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.860334 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:00.977252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:00.977217 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:01.086000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:01.085899 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:02.508153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:02.508120 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:04.575144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:04.575110 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:05.268464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.268435 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:25:05.288057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.288033 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:05.367018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.366073 4433 generic.go:356] "Generic (PLEG): container finished" podID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerID="16b2cfd916aca20744e2346a53c41388dc2242303409953a34489a640e620544" exitCode=0 Oct 30 00:25:05.367018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.366295 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" event={"ID":"64732b06-4a87-41c7-b532-cbf9ca01a0b4","Type":"ContainerDied","Data":"16b2cfd916aca20744e2346a53c41388dc2242303409953a34489a640e620544"} Oct 30 00:25:05.393676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.393554 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:05.699404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.699373 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:25:05.765203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.765178 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:25:05.770872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.770845 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:25:05.800150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800119 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-data\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800185 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-envoy\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800207 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-certs\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800281 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-token\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-token\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800313 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istiod-ca-cert\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800372 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-podinfo\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800409 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-kube-api-access-6c8fl\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-kube-api-access-6c8fl\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800488 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-credential-socket\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.800612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800511 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-data" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.800612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.800603 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-socket\") pod \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\" (UID: \"64732b06-4a87-41c7-b532-cbf9ca01a0b4\") " Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801056 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istiod-ca-cert" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801268 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-socket" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801287 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-certs" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801449 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-credential-socket" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801893 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801915 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801931 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801945 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.803673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.801960 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.804260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.803853 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-podinfo" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:25:05.804260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.804204 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-token" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:05.806231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.806206 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-kube-api-access-6c8fl" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "kube-api-access-6c8fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:05.806428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.806408 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-envoy" pod "64732b06-4a87-41c7-b532-cbf9ca01a0b4" (UID: "64732b06-4a87-41c7-b532-cbf9ca01a0b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.816904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.816411 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" podUID="81d224a3-01d7-49dc-893a-c40b6aeafb32" containerName="payload-processing" containerID="cri-o://6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b" gracePeriod=25 Oct 30 00:25:05.849774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.849693 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" podUID="66607691-eb15-4d08-b2fc-edad9fff498f" containerName="payload-pre-processing" containerID="cri-o://a61f7ad20f8d28bf2a1498825b5a49867a54ae3a491fba9f71ecccd543d19580" gracePeriod=25 Oct 30 00:25:05.902950 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.902911 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-credential-socket\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.903178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903040 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-token\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-token\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.903178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903088 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-certs\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.903288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903194 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-kube-api-access-c9trf\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-kube-api-access-c9trf\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.903288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903239 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-credential-socket\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.903400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903286 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-credential-socket" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.903400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903302 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-envoy\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.903400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903371 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-token\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-token\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.903635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903591 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-kube-api-access-g4bnm\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-kube-api-access-g4bnm\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.903741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903686 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-socket\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.903801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903739 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-data\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.904043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903848 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c90b78a-c0c8-43ee-b682-097031cf6734-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c90b78a-c0c8-43ee-b682-097031cf6734-istiod-ca-cert\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.904043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903920 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-podinfo\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.904043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.903972 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istiod-ca-cert\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.904242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904098 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-certs\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.904242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904172 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-podinfo\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.904242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904229 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-envoy\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.904397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904244 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-socket" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.904397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904291 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-socket\") pod \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\" (UID: \"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf\") " Oct 30 00:25:05.904397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904335 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-data\") pod \"1c90b78a-c0c8-43ee-b682-097031cf6734\" (UID: \"1c90b78a-c0c8-43ee-b682-097031cf6734\") " Oct 30 00:25:05.905031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904629 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-certs" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.905031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.904859 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-certs" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.905186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905072 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-data" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905795 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905829 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905844 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905859 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905872 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8fl\" (UniqueName: \"kubernetes.io/projected/64732b06-4a87-41c7-b532-cbf9ca01a0b4-kube-api-access-6c8fl\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905888 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905903 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.905918 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64732b06-4a87-41c7-b532-cbf9ca01a0b4-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:05.906850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.906628 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-credential-socket" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.906850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.906641 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-data" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.906850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.906674 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c90b78a-c0c8-43ee-b682-097031cf6734-istiod-ca-cert" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:05.907067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.906938 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-socket" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.907556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.907376 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istiod-ca-cert" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:05.907556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.907442 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-envoy" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.911962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.911915 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-envoy" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:05.911962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.911928 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-podinfo" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:25:05.912131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.912079 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-token" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:05.912200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.912160 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-token" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:05.913231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.913209 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-podinfo" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:25:05.913315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.913247 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-kube-api-access-c9trf" pod "1c90b78a-c0c8-43ee-b682-097031cf6734" (UID: "1c90b78a-c0c8-43ee-b682-097031cf6734"). InnerVolumeSpecName "kube-api-access-c9trf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:05.913432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:05.913418 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-kube-api-access-g4bnm" pod "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" (UID: "1b54a68c-7e52-4ed1-8c16-f46783a8a4cf"). InnerVolumeSpecName "kube-api-access-g4bnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017756 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017793 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9trf\" (UniqueName: \"kubernetes.io/projected/1c90b78a-c0c8-43ee-b682-097031cf6734-kube-api-access-c9trf\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017808 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017823 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017836 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017849 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4bnm\" (UniqueName: \"kubernetes.io/projected/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-kube-api-access-g4bnm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017863 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017876 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017889 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c90b78a-c0c8-43ee-b682-097031cf6734-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017901 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017913 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017926 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017938 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c90b78a-c0c8-43ee-b682-097031cf6734-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.018043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.017952 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.187465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.187433 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:25:06.207088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.207052 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:06.332874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.332594 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d224a3-01d7-49dc-893a-c40b6aeafb32-kube-api-access-w8q28\" (UniqueName: \"kubernetes.io/projected/81d224a3-01d7-49dc-893a-c40b6aeafb32-kube-api-access-w8q28\") pod \"81d224a3-01d7-49dc-893a-c40b6aeafb32\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " Oct 30 00:25:06.333967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.333940 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d224a3-01d7-49dc-893a-c40b6aeafb32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d224a3-01d7-49dc-893a-c40b6aeafb32-plugins-config-volume\") pod \"81d224a3-01d7-49dc-893a-c40b6aeafb32\" (UID: \"81d224a3-01d7-49dc-893a-c40b6aeafb32\") " Oct 30 00:25:06.334843 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.334340 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d224a3-01d7-49dc-893a-c40b6aeafb32-plugins-config-volume" pod "81d224a3-01d7-49dc-893a-c40b6aeafb32" (UID: "81d224a3-01d7-49dc-893a-c40b6aeafb32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:06.335968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.335371 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d224a3-01d7-49dc-893a-c40b6aeafb32-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.344639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.344535 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d224a3-01d7-49dc-893a-c40b6aeafb32-kube-api-access-w8q28" pod "81d224a3-01d7-49dc-893a-c40b6aeafb32" (UID: "81d224a3-01d7-49dc-893a-c40b6aeafb32"). InnerVolumeSpecName "kube-api-access-w8q28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:06.389660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.389624 4433 generic.go:356] "Generic (PLEG): container finished" podID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerID="9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741" exitCode=0 Oct 30 00:25:06.389852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.389782 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" Oct 30 00:25:06.390483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.390454 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" event={"ID":"1c90b78a-c0c8-43ee-b682-097031cf6734","Type":"ContainerDied","Data":"9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741"} Oct 30 00:25:06.390720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.390699 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj" event={"ID":"1c90b78a-c0c8-43ee-b682-097031cf6734","Type":"ContainerDied","Data":"00dc8a5d18606ba533bbe1d6c78af7365bd5430545865b760ab8ac98144b015b"} Oct 30 00:25:06.390864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.390849 4433 scope.go:122] "RemoveContainer" containerID="9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741" Oct 30 00:25:06.394378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.393096 4433 generic.go:356] "Generic (PLEG): container finished" podID="66607691-eb15-4d08-b2fc-edad9fff498f" containerID="a61f7ad20f8d28bf2a1498825b5a49867a54ae3a491fba9f71ecccd543d19580" exitCode=0 Oct 30 00:25:06.394378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.393185 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" event={"ID":"66607691-eb15-4d08-b2fc-edad9fff498f","Type":"ContainerDied","Data":"a61f7ad20f8d28bf2a1498825b5a49867a54ae3a491fba9f71ecccd543d19580"} Oct 30 00:25:06.403939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.403908 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" Oct 30 00:25:06.404079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.403980 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57" event={"ID":"64732b06-4a87-41c7-b532-cbf9ca01a0b4","Type":"ContainerDied","Data":"a2b65e343a0eba5e29b30b4954b8cadff15862aa5882247cafd508ae592bd1a6"} Oct 30 00:25:06.407145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.406958 4433 generic.go:356] "Generic (PLEG): container finished" podID="81d224a3-01d7-49dc-893a-c40b6aeafb32" containerID="6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b" exitCode=0 Oct 30 00:25:06.407145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.407004 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" Oct 30 00:25:06.407336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.407002 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" event={"ID":"81d224a3-01d7-49dc-893a-c40b6aeafb32","Type":"ContainerDied","Data":"6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b"} Oct 30 00:25:06.408795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.407613 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb" event={"ID":"81d224a3-01d7-49dc-893a-c40b6aeafb32","Type":"ContainerDied","Data":"bc3e3b07d6029f55d74d247fb3455651d15766dcbeda05f718fefda785aba804"} Oct 30 00:25:06.409439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.409413 4433 generic.go:356] "Generic (PLEG): container finished" podID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerID="dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f" exitCode=0 Oct 30 00:25:06.409623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.409475 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" Oct 30 00:25:06.409907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.409492 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" event={"ID":"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf","Type":"ContainerDied","Data":"dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f"} Oct 30 00:25:06.410036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.410020 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg" event={"ID":"1b54a68c-7e52-4ed1-8c16-f46783a8a4cf","Type":"ContainerDied","Data":"9d163f85cec8200daed6cef462980ae1c69308ca7a491acea79e62c662b7da4a"} Oct 30 00:25:06.418908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.418860 4433 scope.go:122] "RemoveContainer" containerID="9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741" Oct 30 00:25:06.419381 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:06.419247 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741\": container with ID starting with 9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741 not found: ID does not exist" containerID="9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741" Oct 30 00:25:06.419381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.419283 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741"} err="failed to get container status \"9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741\": rpc error: code = NotFound desc = could not find container \"9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741\": container with ID starting with 9ba545b87b417062e4d65d1d66e66119ec7736cd8bad069082b0432df6a6b741 not found: ID does not exist" Oct 30 00:25:06.419381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.419306 4433 scope.go:122] "RemoveContainer" containerID="16b2cfd916aca20744e2346a53c41388dc2242303409953a34489a640e620544" Oct 30 00:25:06.437102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.437072 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8q28\" (UniqueName: \"kubernetes.io/projected/81d224a3-01d7-49dc-893a-c40b6aeafb32-kube-api-access-w8q28\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.471792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.471762 4433 scope.go:122] "RemoveContainer" containerID="6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b" Oct 30 00:25:06.480143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.480082 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj"] Oct 30 00:25:06.486103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.486077 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:25:06.486675 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.486657 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-56xrj"] Oct 30 00:25:06.493389 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.493364 4433 scope.go:122] "RemoveContainer" containerID="6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b" Oct 30 00:25:06.494431 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:06.493793 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b\": container with ID starting with 6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b not found: ID does not exist" containerID="6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b" Oct 30 00:25:06.494431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.493869 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b"} err="failed to get container status \"6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b\": rpc error: code = NotFound desc = could not find container \"6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b\": container with ID starting with 6b4e2dbb225f13b48dfb6ac1e992eb5e230abb4368acdf2f6ce970d4b82aa08b not found: ID does not exist" Oct 30 00:25:06.494431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.493912 4433 scope.go:122] "RemoveContainer" containerID="dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f" Oct 30 00:25:06.510276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.510243 4433 scope.go:122] "RemoveContainer" containerID="dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f" Oct 30 00:25:06.510701 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:06.510671 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f\": container with ID starting with dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f not found: ID does not exist" containerID="dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f" Oct 30 00:25:06.510833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.510710 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f"} err="failed to get container status \"dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f\": rpc error: code = NotFound desc = could not find container \"dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f\": container with ID starting with dc4ed530ef9db80f07fdd38287292cc29b55c56b49f5d7c1f92fef06c7f8d03f not found: ID does not exist" Oct 30 00:25:06.539630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.539521 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66607691-eb15-4d08-b2fc-edad9fff498f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66607691-eb15-4d08-b2fc-edad9fff498f-plugins-config-volume\") pod \"66607691-eb15-4d08-b2fc-edad9fff498f\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " Oct 30 00:25:06.539821 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.539707 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66607691-eb15-4d08-b2fc-edad9fff498f-kube-api-access-wpsbj\" (UniqueName: \"kubernetes.io/projected/66607691-eb15-4d08-b2fc-edad9fff498f-kube-api-access-wpsbj\") pod \"66607691-eb15-4d08-b2fc-edad9fff498f\" (UID: \"66607691-eb15-4d08-b2fc-edad9fff498f\") " Oct 30 00:25:06.540177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.540151 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66607691-eb15-4d08-b2fc-edad9fff498f-plugins-config-volume" pod "66607691-eb15-4d08-b2fc-edad9fff498f" (UID: "66607691-eb15-4d08-b2fc-edad9fff498f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:06.540481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.540458 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66607691-eb15-4d08-b2fc-edad9fff498f-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.549182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.549129 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66607691-eb15-4d08-b2fc-edad9fff498f-kube-api-access-wpsbj" pod "66607691-eb15-4d08-b2fc-edad9fff498f" (UID: "66607691-eb15-4d08-b2fc-edad9fff498f"). InnerVolumeSpecName "kube-api-access-wpsbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:06.553815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.553623 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg"] Oct 30 00:25:06.557537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.557512 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-fhgkg"] Oct 30 00:25:06.591954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.591911 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57"] Oct 30 00:25:06.598719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.598676 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-rtr57"] Oct 30 00:25:06.632459 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.632422 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb"] Oct 30 00:25:06.638383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.638352 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-08dd33-59c49bb788-znxlb"] Oct 30 00:25:06.642164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:06.642137 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsbj\" (UniqueName: \"kubernetes.io/projected/66607691-eb15-4d08-b2fc-edad9fff498f-kube-api-access-wpsbj\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:07.102177 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:07.102137 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:25:07.183400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.183362 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" path="/var/lib/kubelet/pods/1b54a68c-7e52-4ed1-8c16-f46783a8a4cf/volumes" Oct 30 00:25:07.184130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.184111 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" path="/var/lib/kubelet/pods/1c90b78a-c0c8-43ee-b682-097031cf6734/volumes" Oct 30 00:25:07.184613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.184594 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" path="/var/lib/kubelet/pods/64732b06-4a87-41c7-b532-cbf9ca01a0b4/volumes" Oct 30 00:25:07.185320 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.185305 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d224a3-01d7-49dc-893a-c40b6aeafb32" path="/var/lib/kubelet/pods/81d224a3-01d7-49dc-893a-c40b6aeafb32/volumes" Oct 30 00:25:07.420053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.419971 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" Oct 30 00:25:07.420226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.419971 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l" event={"ID":"66607691-eb15-4d08-b2fc-edad9fff498f","Type":"ContainerDied","Data":"dc984adc0f5a2729def4325891cb5a33492f6f7fa4037e0a32e4096c7f7d9553"} Oct 30 00:25:07.420226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.420106 4433 scope.go:122] "RemoveContainer" containerID="a61f7ad20f8d28bf2a1498825b5a49867a54ae3a491fba9f71ecccd543d19580" Oct 30 00:25:07.475297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.475264 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l"] Oct 30 00:25:07.488233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.488200 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08dd33-5798885d5f-cl69l"] Oct 30 00:25:07.961079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:07.961047 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:09.052736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:09.052696 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x"] Oct 30 00:25:09.086003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:09.085970 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28"] Oct 30 00:25:09.185358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:09.185328 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66607691-eb15-4d08-b2fc-edad9fff498f" path="/var/lib/kubelet/pods/66607691-eb15-4d08-b2fc-edad9fff498f/volumes" Oct 30 00:25:09.205290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:09.205256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:09.315197 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:09.315159 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:09.316306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:09.316278 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:25:10.103517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:10.103481 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:11.688458 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:11.688413 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:11.835353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:11.835291 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:11.942887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:11.942797 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:12.163535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:12.163498 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:12.281917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:12.281834 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:12.386218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:12.386179 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:12.823230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:12.823196 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:13.601162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:13.600151 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:14.025029 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.024943 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:14.061817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.061767 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" podUID="04e1172c-4907-4d1b-8276-fda61f96cdef" containerName="payload-processing" containerID="cri-o://2e51be80b9be525fbba7b970cfabc49e5ffb5ca47b0706b6717cdbcfb5c0d4ff" gracePeriod=25 Oct 30 00:25:14.075824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.075769 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7"] Oct 30 00:25:14.076807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.076785 4433 cpu_manager.go:414] "containerMap: removing container" podUID="66607691-eb15-4d08-b2fc-edad9fff498f" containerName="payload-pre-processing" Oct 30 00:25:14.076935 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.076917 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66607691-eb15-4d08-b2fc-edad9fff498f" containerName="payload-pre-processing" Oct 30 00:25:14.077024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.076956 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" Oct 30 00:25:14.077024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.076965 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" Oct 30 00:25:14.077024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.076978 4433 cpu_manager.go:414] "containerMap: removing container" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" Oct 30 00:25:14.077024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.076986 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" Oct 30 00:25:14.077221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077030 4433 cpu_manager.go:414] "containerMap: removing container" podUID="81d224a3-01d7-49dc-893a-c40b6aeafb32" containerName="payload-processing" Oct 30 00:25:14.077221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077038 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d224a3-01d7-49dc-893a-c40b6aeafb32" containerName="payload-processing" Oct 30 00:25:14.077221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077069 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" Oct 30 00:25:14.077221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077077 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" Oct 30 00:25:14.077437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077268 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b54a68c-7e52-4ed1-8c16-f46783a8a4cf" containerName="istio-proxy" Oct 30 00:25:14.077437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077291 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="66607691-eb15-4d08-b2fc-edad9fff498f" containerName="payload-pre-processing" Oct 30 00:25:14.077437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077305 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c90b78a-c0c8-43ee-b682-097031cf6734" containerName="istio-proxy" Oct 30 00:25:14.077437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077315 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="64732b06-4a87-41c7-b532-cbf9ca01a0b4" containerName="istio-proxy" Oct 30 00:25:14.077437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.077330 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d224a3-01d7-49dc-893a-c40b6aeafb32" containerName="payload-processing" Oct 30 00:25:14.080827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.080759 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.086652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.086615 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" podUID="c90dfe48-38f9-463a-aecc-887fa1d2c6e0" containerName="payload-pre-processing" containerID="cri-o://655421f3bae80b61c90a32f8e05bd1ed34a0b011fc1a5c0ddcc06ed959bbf915" gracePeriod=25 Oct 30 00:25:14.099245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.099205 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7"] Oct 30 00:25:14.105154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.104725 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f8397b-serving-cert\"" Oct 30 00:25:14.140274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.139261 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:14.184129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.184095 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.184340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.184169 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0df4b3-3098-41d8-84ca-23c5a23b3183-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.184340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.184273 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bt4\" (UniqueName: \"kubernetes.io/projected/cd0df4b3-3098-41d8-84ca-23c5a23b3183-kube-api-access-g2bt4\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.184478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.184366 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-metrics-certs\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.257921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.253243 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:14.290463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.290065 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.290463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.290148 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0df4b3-3098-41d8-84ca-23c5a23b3183-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.290463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.290275 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bt4\" (UniqueName: \"kubernetes.io/projected/cd0df4b3-3098-41d8-84ca-23c5a23b3183-kube-api-access-g2bt4\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.293429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.290796 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-metrics-certs\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.293429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.292915 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0df4b3-3098-41d8-84ca-23c5a23b3183-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.299130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.299039 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.332646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.310013 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-metrics-certs\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.335843 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.335777 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bt4\" (UniqueName: \"kubernetes.io/projected/cd0df4b3-3098-41d8-84ca-23c5a23b3183-kube-api-access-g2bt4\") pod \"maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.381053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.376465 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:14.566647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.536424 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:14.629941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.624728 4433 generic.go:356] "Generic (PLEG): container finished" podID="c90dfe48-38f9-463a-aecc-887fa1d2c6e0" containerID="655421f3bae80b61c90a32f8e05bd1ed34a0b011fc1a5c0ddcc06ed959bbf915" exitCode=0 Oct 30 00:25:14.629941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.624955 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" event={"ID":"c90dfe48-38f9-463a-aecc-887fa1d2c6e0","Type":"ContainerDied","Data":"655421f3bae80b61c90a32f8e05bd1ed34a0b011fc1a5c0ddcc06ed959bbf915"} Oct 30 00:25:14.629941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.627124 4433 generic.go:356] "Generic (PLEG): container finished" podID="04e1172c-4907-4d1b-8276-fda61f96cdef" containerID="2e51be80b9be525fbba7b970cfabc49e5ffb5ca47b0706b6717cdbcfb5c0d4ff" exitCode=0 Oct 30 00:25:14.629941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.627276 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" event={"ID":"04e1172c-4907-4d1b-8276-fda61f96cdef","Type":"ContainerDied","Data":"2e51be80b9be525fbba7b970cfabc49e5ffb5ca47b0706b6717cdbcfb5c0d4ff"} Oct 30 00:25:14.652708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.652684 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:25:14.677800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.674105 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:14.713321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.712398 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:25:14.744440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.744375 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-kube-api-access-g4x6n\" (UniqueName: \"kubernetes.io/projected/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-kube-api-access-g4x6n\") pod \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " Oct 30 00:25:14.744814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.744795 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e1172c-4907-4d1b-8276-fda61f96cdef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1172c-4907-4d1b-8276-fda61f96cdef-plugins-config-volume\") pod \"04e1172c-4907-4d1b-8276-fda61f96cdef\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " Oct 30 00:25:14.745053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.745041 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e1172c-4907-4d1b-8276-fda61f96cdef-kube-api-access-vvcg4\" (UniqueName: \"kubernetes.io/projected/04e1172c-4907-4d1b-8276-fda61f96cdef-kube-api-access-vvcg4\") pod \"04e1172c-4907-4d1b-8276-fda61f96cdef\" (UID: \"04e1172c-4907-4d1b-8276-fda61f96cdef\") " Oct 30 00:25:14.745358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.745343 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-plugins-config-volume\") pod \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\" (UID: \"c90dfe48-38f9-463a-aecc-887fa1d2c6e0\") " Oct 30 00:25:14.757686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.750829 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-plugins-config-volume" pod "c90dfe48-38f9-463a-aecc-887fa1d2c6e0" (UID: "c90dfe48-38f9-463a-aecc-887fa1d2c6e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:14.764362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.764304 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-kube-api-access-g4x6n" pod "c90dfe48-38f9-463a-aecc-887fa1d2c6e0" (UID: "c90dfe48-38f9-463a-aecc-887fa1d2c6e0"). InnerVolumeSpecName "kube-api-access-g4x6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:14.770596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.770542 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e1172c-4907-4d1b-8276-fda61f96cdef-plugins-config-volume" pod "04e1172c-4907-4d1b-8276-fda61f96cdef" (UID: "04e1172c-4907-4d1b-8276-fda61f96cdef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:14.829672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.824143 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e1172c-4907-4d1b-8276-fda61f96cdef-kube-api-access-vvcg4" pod "04e1172c-4907-4d1b-8276-fda61f96cdef" (UID: "04e1172c-4907-4d1b-8276-fda61f96cdef"). InnerVolumeSpecName "kube-api-access-vvcg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:14.849251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.849152 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1172c-4907-4d1b-8276-fda61f96cdef-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:14.849251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.849194 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvcg4\" (UniqueName: \"kubernetes.io/projected/04e1172c-4907-4d1b-8276-fda61f96cdef-kube-api-access-vvcg4\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:14.849251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.849210 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:14.849251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:14.849224 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4x6n\" (UniqueName: \"kubernetes.io/projected/c90dfe48-38f9-463a-aecc-887fa1d2c6e0-kube-api-access-g4x6n\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:15.009168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.009139 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7"] Oct 30 00:25:15.010403 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:25:15.010374 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0df4b3_3098_41d8_84ca_23c5a23b3183.slice/crio-fac4a78ff3461f8c15a6f3be8ddc2300220334f3fd6fa97e1add97584193b044 WatchSource:0}: Error finding container fac4a78ff3461f8c15a6f3be8ddc2300220334f3fd6fa97e1add97584193b044: Status 404 returned error can't find the container with id fac4a78ff3461f8c15a6f3be8ddc2300220334f3fd6fa97e1add97584193b044 Oct 30 00:25:15.025988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.025951 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:15.638956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.638912 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" Oct 30 00:25:15.639112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.638929 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28" event={"ID":"c90dfe48-38f9-463a-aecc-887fa1d2c6e0","Type":"ContainerDied","Data":"1f892d272fe18ea57e55b4929f7ccba8482139ec2d82f8e854a306e036368b3c"} Oct 30 00:25:15.639205 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.639141 4433 scope.go:122] "RemoveContainer" containerID="655421f3bae80b61c90a32f8e05bd1ed34a0b011fc1a5c0ddcc06ed959bbf915" Oct 30 00:25:15.641228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.641198 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" event={"ID":"cd0df4b3-3098-41d8-84ca-23c5a23b3183","Type":"ContainerStarted","Data":"b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0"} Oct 30 00:25:15.641361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.641250 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" event={"ID":"cd0df4b3-3098-41d8-84ca-23c5a23b3183","Type":"ContainerStarted","Data":"fac4a78ff3461f8c15a6f3be8ddc2300220334f3fd6fa97e1add97584193b044"} Oct 30 00:25:15.641361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.641309 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:15.643321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.643301 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" event={"ID":"04e1172c-4907-4d1b-8276-fda61f96cdef","Type":"ContainerDied","Data":"48fb72244a8d793f80e7c8d0fd2db19303ed44a1e54349b0f54860bea228b445"} Oct 30 00:25:15.643424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.643321 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x" Oct 30 00:25:15.652375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.652344 4433 scope.go:122] "RemoveContainer" containerID="2e51be80b9be525fbba7b970cfabc49e5ffb5ca47b0706b6717cdbcfb5c0d4ff" Oct 30 00:25:15.695012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.694979 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28"] Oct 30 00:25:15.707896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.707832 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0aea47-65697d64cf-z2d28"] Oct 30 00:25:15.746872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.746821 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" podStartSLOduration=1.306611371 podStartE2EDuration="1.746807693s" totalImagesPullingTime="440.196322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:25:15.74555478 +0000 UTC m=+1224.679623315" watchObservedRunningTime="2026-10-30 00:25:15.746807693 +0000 UTC m=+1224.680876227" Oct 30 00:25:15.785591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.785536 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x"] Oct 30 00:25:15.789614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.789591 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0aea47-6ff544fb-zwr2x"] Oct 30 00:25:15.881803 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:15.881769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:17.189550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:17.189519 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e1172c-4907-4d1b-8276-fda61f96cdef" path="/var/lib/kubelet/pods/04e1172c-4907-4d1b-8276-fda61f96cdef/volumes" Oct 30 00:25:17.190310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:17.190190 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90dfe48-38f9-463a-aecc-887fa1d2c6e0" path="/var/lib/kubelet/pods/c90dfe48-38f9-463a-aecc-887fa1d2c6e0/volumes" Oct 30 00:25:17.393942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:17.393909 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:18.841475 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:18.841440 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:18.945818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:18.945784 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:19.209669 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:19.208897 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:19.319381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:19.319339 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:19.649978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:19.649946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:19.760888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:19.760843 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:20.554545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:20.554499 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:21.658751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:21.658720 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:25:22.120200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:22.120156 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:23.499372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:23.499321 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:24.768246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:24.768207 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:24.877376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:24.877323 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:24.989955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:24.989909 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:25.256070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:25.256030 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:25.385911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:25.385865 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:25.495469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:25.495406 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:25.924409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:25.924326 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:26.941960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:26.941928 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:28.439015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:28.438974 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:30.309901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:30.309864 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:32.120737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:32.120703 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:32.311651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:32.311617 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:32.417290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:32.417209 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:32.753698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:32.753612 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr"] Oct 30 00:25:32.753942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:32.753911 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" containerID="cri-o://03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b" gracePeriod=30 Oct 30 00:25:32.783610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:32.782929 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:33.697907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:33.697869 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:35.143773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:35.143734 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:36.564078 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:36.564041 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:36.890892 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:36.890809 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w"] Oct 30 00:25:37.040239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:37.040195 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:37.101322 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:37.101290 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:25:37.157367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:37.157272 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:37.265517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:37.265486 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:38.044658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.044597 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:25:38.065148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.065109 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:38.142414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.142376 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-certs\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.142708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.142430 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-socket\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.142708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.142470 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-data\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.142708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.142505 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-podinfo\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.142708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.142685 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istiod-ca-cert\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.142982 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.142801 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-token\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-token\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.143101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.142988 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-credential-socket\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.143188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.143129 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-kube-api-access-hdwb4\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-kube-api-access-hdwb4\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.143266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.143198 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-envoy\") pod \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\" (UID: \"73c6a0e5-d146-4014-9851-0cb3734cc5f0\") " Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.146129 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-podinfo" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.146380 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-certs" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.146715 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-socket" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.146918 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-data" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.147455 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.147477 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.147490 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.147505 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.150811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.150541 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-credential-socket" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.152310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.152284 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istiod-ca-cert" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:38.154171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.154145 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-token" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:38.154848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.154823 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-kube-api-access-hdwb4" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "kube-api-access-hdwb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:38.157122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.156538 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-envoy" pod "73c6a0e5-d146-4014-9851-0cb3734cc5f0" (UID: "73c6a0e5-d146-4014-9851-0cb3734cc5f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.172536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.172499 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:38.249863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.249832 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.249863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.249862 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.249863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.249872 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.250144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.249886 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdwb4\" (UniqueName: \"kubernetes.io/projected/73c6a0e5-d146-4014-9851-0cb3734cc5f0-kube-api-access-hdwb4\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.250144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.249901 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c6a0e5-d146-4014-9851-0cb3734cc5f0-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.849105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.849063 4433 generic.go:356] "Generic (PLEG): container finished" podID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerID="03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b" exitCode=0 Oct 30 00:25:38.849308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.849195 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" Oct 30 00:25:38.849308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.849220 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" event={"ID":"73c6a0e5-d146-4014-9851-0cb3734cc5f0","Type":"ContainerDied","Data":"03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b"} Oct 30 00:25:38.849308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.849281 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr" event={"ID":"73c6a0e5-d146-4014-9851-0cb3734cc5f0","Type":"ContainerDied","Data":"89d8667bfe11bda78fb0beae3e112182f23468dffe91fcc8ae2f4f74d8f163d5"} Oct 30 00:25:38.849308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.849308 4433 scope.go:122] "RemoveContainer" containerID="03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b" Oct 30 00:25:38.869652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.869625 4433 scope.go:122] "RemoveContainer" containerID="03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b" Oct 30 00:25:38.870011 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:38.869987 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b\": container with ID starting with 03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b not found: ID does not exist" containerID="03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b" Oct 30 00:25:38.870106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.870018 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b"} err="failed to get container status \"03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b\": rpc error: code = NotFound desc = could not find container \"03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b\": container with ID starting with 03187c130b906450cc461fc5ff92bbbe41cf2f0eccb23148af05fafcdaffb14b not found: ID does not exist" Oct 30 00:25:38.919472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.919436 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr"] Oct 30 00:25:38.932351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:38.932306 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-26nnr"] Oct 30 00:25:39.181153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.181076 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" path="/var/lib/kubelet/pods/73c6a0e5-d146-4014-9851-0cb3734cc5f0/volumes" Oct 30 00:25:39.377861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.377823 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng"] Oct 30 00:25:39.378171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.378116 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" podUID="d9c76eba-4a75-4417-a6f4-21b297c54ac0" containerName="maas-api" containerID="cri-o://799bdc1e85f824e02470f47312ca6ba216caa931045ab6f3daa280bd7b7f9a91" gracePeriod=30 Oct 30 00:25:39.441587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.439914 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:39.884328 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.884281 4433 generic.go:356] "Generic (PLEG): container finished" podID="d9c76eba-4a75-4417-a6f4-21b297c54ac0" containerID="799bdc1e85f824e02470f47312ca6ba216caa931045ab6f3daa280bd7b7f9a91" exitCode=0 Oct 30 00:25:39.884611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.884560 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" event={"ID":"d9c76eba-4a75-4417-a6f4-21b297c54ac0","Type":"ContainerDied","Data":"799bdc1e85f824e02470f47312ca6ba216caa931045ab6f3daa280bd7b7f9a91"} Oct 30 00:25:39.884703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.884685 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" event={"ID":"d9c76eba-4a75-4417-a6f4-21b297c54ac0","Type":"ContainerDied","Data":"2058e11c27a9577c7b68045ba3bf8a10a1288e2793f5cbdf65409c500456675f"} Oct 30 00:25:39.884792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.884716 4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2058e11c27a9577c7b68045ba3bf8a10a1288e2793f5cbdf65409c500456675f" Oct 30 00:25:39.887104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.887081 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:25:39.931902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.931868 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-maas-api-tls\") pod \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " Oct 30 00:25:39.932105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.931965 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c76eba-4a75-4417-a6f4-21b297c54ac0-kube-api-access-tjkpv\" (UniqueName: \"kubernetes.io/projected/d9c76eba-4a75-4417-a6f4-21b297c54ac0-kube-api-access-tjkpv\") pod \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " Oct 30 00:25:39.932105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.932036 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-metrics-certs\") pod \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " Oct 30 00:25:39.932241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.932113 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c76eba-4a75-4417-a6f4-21b297c54ac0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c76eba-4a75-4417-a6f4-21b297c54ac0-openshift-service-ca\") pod \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\" (UID: \"d9c76eba-4a75-4417-a6f4-21b297c54ac0\") " Oct 30 00:25:39.933828 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.933792 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c76eba-4a75-4417-a6f4-21b297c54ac0-openshift-service-ca" pod "d9c76eba-4a75-4417-a6f4-21b297c54ac0" (UID: "d9c76eba-4a75-4417-a6f4-21b297c54ac0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:39.934862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.934833 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c76eba-4a75-4417-a6f4-21b297c54ac0-kube-api-access-tjkpv" pod "d9c76eba-4a75-4417-a6f4-21b297c54ac0" (UID: "d9c76eba-4a75-4417-a6f4-21b297c54ac0"). InnerVolumeSpecName "kube-api-access-tjkpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:39.935004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.934974 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-maas-api-tls" pod "d9c76eba-4a75-4417-a6f4-21b297c54ac0" (UID: "d9c76eba-4a75-4417-a6f4-21b297c54ac0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:39.935811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.935777 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-metrics-certs" pod "d9c76eba-4a75-4417-a6f4-21b297c54ac0" (UID: "d9c76eba-4a75-4417-a6f4-21b297c54ac0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:39.990962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:39.990923 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:40.035407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.035332 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkpv\" (UniqueName: \"kubernetes.io/projected/d9c76eba-4a75-4417-a6f4-21b297c54ac0-kube-api-access-tjkpv\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:40.035705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.035684 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:40.035824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.035812 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c76eba-4a75-4417-a6f4-21b297c54ac0-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:40.035919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.035897 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c76eba-4a75-4417-a6f4-21b297c54ac0-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:40.097768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.097730 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:40.205545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.205506 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:40.713498 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.713390 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:40.820946 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.820914 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:40.896002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.895972 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng" Oct 30 00:25:40.956618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.956587 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng"] Oct 30 00:25:40.969813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.969724 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0aea47-67c95967b5-zknng"] Oct 30 00:25:40.990615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:40.990585 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt"] Oct 30 00:25:41.141381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:41.133464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:41.181304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:41.181276 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c76eba-4a75-4417-a6f4-21b297c54ac0" path="/var/lib/kubelet/pods/d9c76eba-4a75-4417-a6f4-21b297c54ac0/volumes" Oct 30 00:25:41.241436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:41.241347 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:41.891502 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:41.891461 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" podUID="fba9c019-43cf-4e31-a9ad-1a887c2be5ff" containerName="payload-processing" containerID="cri-o://a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9" gracePeriod=25 Oct 30 00:25:41.904593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:41.902402 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:42.215324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.215299 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:25:42.280122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.280073 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-plugins-config-volume\") pod \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " Oct 30 00:25:42.280642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.280140 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-kube-api-access-kdmb9\" (UniqueName: \"kubernetes.io/projected/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-kube-api-access-kdmb9\") pod \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\" (UID: \"fba9c019-43cf-4e31-a9ad-1a887c2be5ff\") " Oct 30 00:25:42.280642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.280531 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-plugins-config-volume" pod "fba9c019-43cf-4e31-a9ad-1a887c2be5ff" (UID: "fba9c019-43cf-4e31-a9ad-1a887c2be5ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:42.280963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.280941 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:42.282421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.282398 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-kube-api-access-kdmb9" pod "fba9c019-43cf-4e31-a9ad-1a887c2be5ff" (UID: "fba9c019-43cf-4e31-a9ad-1a887c2be5ff"). InnerVolumeSpecName "kube-api-access-kdmb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:42.383045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.382999 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdmb9\" (UniqueName: \"kubernetes.io/projected/fba9c019-43cf-4e31-a9ad-1a887c2be5ff-kube-api-access-kdmb9\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:42.918020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.917985 4433 generic.go:356] "Generic (PLEG): container finished" podID="fba9c019-43cf-4e31-a9ad-1a887c2be5ff" containerID="a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9" exitCode=0 Oct 30 00:25:42.918218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.918039 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" event={"ID":"fba9c019-43cf-4e31-a9ad-1a887c2be5ff","Type":"ContainerDied","Data":"a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9"} Oct 30 00:25:42.918218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.918045 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" Oct 30 00:25:42.918218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.918104 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w" event={"ID":"fba9c019-43cf-4e31-a9ad-1a887c2be5ff","Type":"ContainerDied","Data":"903c321387261500deb0baeda67b1618dd8c1423bdf1cadbc0e2ed7a4553306a"} Oct 30 00:25:42.918218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.918126 4433 scope.go:122] "RemoveContainer" containerID="a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9" Oct 30 00:25:42.928982 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.928963 4433 scope.go:122] "RemoveContainer" containerID="a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9" Oct 30 00:25:42.929253 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:42.929232 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9\": container with ID starting with a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9 not found: ID does not exist" containerID="a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9" Oct 30 00:25:42.929302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.929262 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9"} err="failed to get container status \"a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9\": rpc error: code = NotFound desc = could not find container \"a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9\": container with ID starting with a9e017abdf4e4e61e9957300dfd52823a58096a22eaeec90f0c3113c4104d9d9 not found: ID does not exist" Oct 30 00:25:42.973595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.973552 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w"] Oct 30 00:25:42.986443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:42.986414 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f8397b-57d986f5b6-pgp6w"] Oct 30 00:25:43.182509 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:43.182431 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba9c019-43cf-4e31-a9ad-1a887c2be5ff" path="/var/lib/kubelet/pods/fba9c019-43cf-4e31-a9ad-1a887c2be5ff/volumes" Oct 30 00:25:45.991884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:45.991841 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" podUID="b2c47144-8118-490f-82ec-e2ee06a52b7c" containerName="payload-processing" containerID="cri-o://f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e" gracePeriod=25 Oct 30 00:25:46.283185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.283159 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:25:46.446375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.446335 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c47144-8118-490f-82ec-e2ee06a52b7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47144-8118-490f-82ec-e2ee06a52b7c-plugins-config-volume\") pod \"b2c47144-8118-490f-82ec-e2ee06a52b7c\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " Oct 30 00:25:46.446635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.446616 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c47144-8118-490f-82ec-e2ee06a52b7c-kube-api-access-gwdzm\" (UniqueName: \"kubernetes.io/projected/b2c47144-8118-490f-82ec-e2ee06a52b7c-kube-api-access-gwdzm\") pod \"b2c47144-8118-490f-82ec-e2ee06a52b7c\" (UID: \"b2c47144-8118-490f-82ec-e2ee06a52b7c\") " Oct 30 00:25:46.446747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.446716 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c47144-8118-490f-82ec-e2ee06a52b7c-plugins-config-volume" pod "b2c47144-8118-490f-82ec-e2ee06a52b7c" (UID: "b2c47144-8118-490f-82ec-e2ee06a52b7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:46.447200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.447180 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47144-8118-490f-82ec-e2ee06a52b7c-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:46.448776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.448756 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c47144-8118-490f-82ec-e2ee06a52b7c-kube-api-access-gwdzm" pod "b2c47144-8118-490f-82ec-e2ee06a52b7c" (UID: "b2c47144-8118-490f-82ec-e2ee06a52b7c"). InnerVolumeSpecName "kube-api-access-gwdzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:46.548491 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.548400 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwdzm\" (UniqueName: \"kubernetes.io/projected/b2c47144-8118-490f-82ec-e2ee06a52b7c-kube-api-access-gwdzm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:46.904658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.904618 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:46.954974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.954930 4433 generic.go:356] "Generic (PLEG): container finished" podID="b2c47144-8118-490f-82ec-e2ee06a52b7c" containerID="f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e" exitCode=0 Oct 30 00:25:46.955271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.955228 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" event={"ID":"b2c47144-8118-490f-82ec-e2ee06a52b7c","Type":"ContainerDied","Data":"f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e"} Oct 30 00:25:46.955356 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.955282 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" Oct 30 00:25:46.955356 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.955299 4433 scope.go:122] "RemoveContainer" containerID="f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e" Oct 30 00:25:46.955649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.955285 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt" event={"ID":"b2c47144-8118-490f-82ec-e2ee06a52b7c","Type":"ContainerDied","Data":"a44b5004fa680e6d6e2205d513bd2acc544aee5c2e8c64f01f4f362befc1c37e"} Oct 30 00:25:46.972309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.972283 4433 scope.go:122] "RemoveContainer" containerID="f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e" Oct 30 00:25:46.972689 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:46.972662 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e\": container with ID starting with f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e not found: ID does not exist" containerID="f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e" Oct 30 00:25:46.972761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:46.972699 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e"} err="failed to get container status \"f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e\": rpc error: code = NotFound desc = could not find container \"f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e\": container with ID starting with f25a7d6ee3be25ee76911da7d957d979efbcf8a1d755cbd2bb40720d402bd77e not found: ID does not exist" Oct 30 00:25:47.010854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:47.010817 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt"] Oct 30 00:25:47.023345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:47.023310 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb3ad-658768ff4b-xmjzt"] Oct 30 00:25:47.077106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:47.076913 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:47.184642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:47.184550 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c47144-8118-490f-82ec-e2ee06a52b7c" path="/var/lib/kubelet/pods/b2c47144-8118-490f-82ec-e2ee06a52b7c/volumes" Oct 30 00:25:47.252546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:47.252508 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:52.900777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:52.900738 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:54.287097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:54.287054 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm"] Oct 30 00:25:54.287684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:54.287323 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" podUID="5ee60444-e0e5-46cb-81fc-9522041a5b22" containerName="maas-api" containerID="cri-o://86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637" gracePeriod=30 Oct 30 00:25:54.314876 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:54.313772 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:54.822228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:54.822105 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:54.946861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:54.946827 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:25:55.026410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.026212 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:55.034383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.034261 4433 generic.go:356] "Generic (PLEG): container finished" podID="5ee60444-e0e5-46cb-81fc-9522041a5b22" containerID="86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637" exitCode=0 Oct 30 00:25:55.034383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.034306 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" Oct 30 00:25:55.034780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.034396 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" event={"ID":"5ee60444-e0e5-46cb-81fc-9522041a5b22","Type":"ContainerDied","Data":"86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637"} Oct 30 00:25:55.034780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.034462 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm" event={"ID":"5ee60444-e0e5-46cb-81fc-9522041a5b22","Type":"ContainerDied","Data":"36c93e6b23289d91d0f3ab19b06cdb4391bde48a0de17897f55321f2d1858cec"} Oct 30 00:25:55.034780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.034486 4433 scope.go:122] "RemoveContainer" containerID="86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637" Oct 30 00:25:55.045982 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.045958 4433 scope.go:122] "RemoveContainer" containerID="86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637" Oct 30 00:25:55.046317 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:55.046291 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637\": container with ID starting with 86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637 not found: ID does not exist" containerID="86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637" Oct 30 00:25:55.046404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.046325 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637"} err="failed to get container status \"86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637\": rpc error: code = NotFound desc = could not find container \"86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637\": container with ID starting with 86285bc2b7eaf482e7d8996cbbec680632027e620641ddf2f043bff6223e0637 not found: ID does not exist" Oct 30 00:25:55.113658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.113548 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-maas-api-tls\") pod \"5ee60444-e0e5-46cb-81fc-9522041a5b22\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " Oct 30 00:25:55.114160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.114111 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-metrics-certs\") pod \"5ee60444-e0e5-46cb-81fc-9522041a5b22\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " Oct 30 00:25:55.114528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.114504 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee60444-e0e5-46cb-81fc-9522041a5b22-kube-api-access-sjvxw\" (UniqueName: \"kubernetes.io/projected/5ee60444-e0e5-46cb-81fc-9522041a5b22-kube-api-access-sjvxw\") pod \"5ee60444-e0e5-46cb-81fc-9522041a5b22\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " Oct 30 00:25:55.114989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.114948 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee60444-e0e5-46cb-81fc-9522041a5b22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee60444-e0e5-46cb-81fc-9522041a5b22-openshift-service-ca\") pod \"5ee60444-e0e5-46cb-81fc-9522041a5b22\" (UID: \"5ee60444-e0e5-46cb-81fc-9522041a5b22\") " Oct 30 00:25:55.118044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.118016 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee60444-e0e5-46cb-81fc-9522041a5b22-openshift-service-ca" pod "5ee60444-e0e5-46cb-81fc-9522041a5b22" (UID: "5ee60444-e0e5-46cb-81fc-9522041a5b22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:55.121070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.121044 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee60444-e0e5-46cb-81fc-9522041a5b22-kube-api-access-sjvxw" pod "5ee60444-e0e5-46cb-81fc-9522041a5b22" (UID: "5ee60444-e0e5-46cb-81fc-9522041a5b22"). InnerVolumeSpecName "kube-api-access-sjvxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:55.123449 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.123428 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-metrics-certs" pod "5ee60444-e0e5-46cb-81fc-9522041a5b22" (UID: "5ee60444-e0e5-46cb-81fc-9522041a5b22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:55.130028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.129990 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-maas-api-tls" pod "5ee60444-e0e5-46cb-81fc-9522041a5b22" (UID: "5ee60444-e0e5-46cb-81fc-9522041a5b22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:55.138805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.138404 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:55.219832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.219799 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee60444-e0e5-46cb-81fc-9522041a5b22-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:55.219832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.219837 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:55.220067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.219848 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee60444-e0e5-46cb-81fc-9522041a5b22-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:55.220067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.219856 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvxw\" (UniqueName: \"kubernetes.io/projected/5ee60444-e0e5-46cb-81fc-9522041a5b22-kube-api-access-sjvxw\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:55.386084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.385978 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm"] Oct 30 00:25:55.390391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.390357 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08dd33-769fd9b55-9gffm"] Oct 30 00:25:55.588736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:55.588701 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:56.744221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:56.744188 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:57.068843 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:57.068812 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:57.183380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:57.183338 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee60444-e0e5-46cb-81fc-9522041a5b22" path="/var/lib/kubelet/pods/5ee60444-e0e5-46cb-81fc-9522041a5b22/volumes" Oct 30 00:25:57.586159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:57.586123 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.042541 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.042317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.202640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.202584 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.308117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.308081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.354285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.351748 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp"] Oct 30 00:25:58.354285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.352354 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" podUID="2d30c41b-626d-4e4b-8d0b-5211b589275d" containerName="maas-api" containerID="cri-o://a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350" gracePeriod=30 Oct 30 00:25:58.421203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.421167 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.539533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.538176 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f"] Oct 30 00:25:58.541149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.540415 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" podUID="1cd4cf6e-0a68-44d4-959e-087bd8b443af" containerName="payload-pre-processing" containerID="cri-o://11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7" gracePeriod=30 Oct 30 00:25:58.565069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.564948 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.678104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.678033 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.860089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.829783 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:25:58.860089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.854709 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:58.944937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.943451 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d30c41b-626d-4e4b-8d0b-5211b589275d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d30c41b-626d-4e4b-8d0b-5211b589275d-openshift-service-ca\") pod \"2d30c41b-626d-4e4b-8d0b-5211b589275d\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " Oct 30 00:25:58.944937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.943505 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d30c41b-626d-4e4b-8d0b-5211b589275d-kube-api-access-q5bnl\" (UniqueName: \"kubernetes.io/projected/2d30c41b-626d-4e4b-8d0b-5211b589275d-kube-api-access-q5bnl\") pod \"2d30c41b-626d-4e4b-8d0b-5211b589275d\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " Oct 30 00:25:58.944937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.943655 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-metrics-certs\") pod \"2d30c41b-626d-4e4b-8d0b-5211b589275d\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " Oct 30 00:25:58.944937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.943749 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-maas-api-tls\") pod \"2d30c41b-626d-4e4b-8d0b-5211b589275d\" (UID: \"2d30c41b-626d-4e4b-8d0b-5211b589275d\") " Oct 30 00:25:58.946067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.945792 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d30c41b-626d-4e4b-8d0b-5211b589275d-openshift-service-ca" pod "2d30c41b-626d-4e4b-8d0b-5211b589275d" (UID: "2d30c41b-626d-4e4b-8d0b-5211b589275d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:58.949087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.949032 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d30c41b-626d-4e4b-8d0b-5211b589275d-kube-api-access-q5bnl" pod "2d30c41b-626d-4e4b-8d0b-5211b589275d" (UID: "2d30c41b-626d-4e4b-8d0b-5211b589275d"). InnerVolumeSpecName "kube-api-access-q5bnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:58.969774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.969739 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-maas-api-tls" pod "2d30c41b-626d-4e4b-8d0b-5211b589275d" (UID: "2d30c41b-626d-4e4b-8d0b-5211b589275d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:58.971273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.971226 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-metrics-certs" pod "2d30c41b-626d-4e4b-8d0b-5211b589275d" (UID: "2d30c41b-626d-4e4b-8d0b-5211b589275d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:58.985815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:58.984265 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:25:59.016419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.016387 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:59.045682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.045646 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd4cf6e-0a68-44d4-959e-087bd8b443af-kube-api-access-xbjhb\" (UniqueName: \"kubernetes.io/projected/1cd4cf6e-0a68-44d4-959e-087bd8b443af-kube-api-access-xbjhb\") pod \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " Oct 30 00:25:59.046187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.045760 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd4cf6e-0a68-44d4-959e-087bd8b443af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd4cf6e-0a68-44d4-959e-087bd8b443af-plugins-config-volume\") pod \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\" (UID: \"1cd4cf6e-0a68-44d4-959e-087bd8b443af\") " Oct 30 00:25:59.046809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.046780 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d30c41b-626d-4e4b-8d0b-5211b589275d-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.046809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.046810 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5bnl\" (UniqueName: \"kubernetes.io/projected/2d30c41b-626d-4e4b-8d0b-5211b589275d-kube-api-access-q5bnl\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.047134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.046823 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.047134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.046835 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d30c41b-626d-4e4b-8d0b-5211b589275d-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.047790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.047766 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd4cf6e-0a68-44d4-959e-087bd8b443af-plugins-config-volume" pod "1cd4cf6e-0a68-44d4-959e-087bd8b443af" (UID: "1cd4cf6e-0a68-44d4-959e-087bd8b443af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:59.051212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.051155 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd4cf6e-0a68-44d4-959e-087bd8b443af-kube-api-access-xbjhb" pod "1cd4cf6e-0a68-44d4-959e-087bd8b443af" (UID: "1cd4cf6e-0a68-44d4-959e-087bd8b443af"). InnerVolumeSpecName "kube-api-access-xbjhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:59.090215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.089424 4433 generic.go:356] "Generic (PLEG): container finished" podID="2d30c41b-626d-4e4b-8d0b-5211b589275d" containerID="a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350" exitCode=0 Oct 30 00:25:59.090215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.089515 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" event={"ID":"2d30c41b-626d-4e4b-8d0b-5211b589275d","Type":"ContainerDied","Data":"a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350"} Oct 30 00:25:59.090215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.089614 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" event={"ID":"2d30c41b-626d-4e4b-8d0b-5211b589275d","Type":"ContainerDied","Data":"c40257571dc8b98a9428eb08adcfd8e8c300f5ef8b20c0404ff6a190b15088d5"} Oct 30 00:25:59.090215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.089648 4433 scope.go:122] "RemoveContainer" containerID="a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350" Oct 30 00:25:59.090215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.089792 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp" Oct 30 00:25:59.101504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.100404 4433 generic.go:356] "Generic (PLEG): container finished" podID="1cd4cf6e-0a68-44d4-959e-087bd8b443af" containerID="11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7" exitCode=0 Oct 30 00:25:59.101504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.100741 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" event={"ID":"1cd4cf6e-0a68-44d4-959e-087bd8b443af","Type":"ContainerDied","Data":"11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7"} Oct 30 00:25:59.101504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.100803 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" event={"ID":"1cd4cf6e-0a68-44d4-959e-087bd8b443af","Type":"ContainerDied","Data":"327d6b50fd2e70fc40bf98ec44971e8e4643392ed886c71ebb831a9f2cf04912"} Oct 30 00:25:59.101504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.100905 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f" Oct 30 00:25:59.135677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.130263 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:59.149274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.149243 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbjhb\" (UniqueName: \"kubernetes.io/projected/1cd4cf6e-0a68-44d4-959e-087bd8b443af-kube-api-access-xbjhb\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.149274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.149278 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd4cf6e-0a68-44d4-959e-087bd8b443af-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.154659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.154479 4433 scope.go:122] "RemoveContainer" containerID="a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350" Oct 30 00:25:59.155108 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:59.155083 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350\": container with ID starting with a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350 not found: ID does not exist" containerID="a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350" Oct 30 00:25:59.155212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.155116 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350"} err="failed to get container status \"a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350\": rpc error: code = NotFound desc = could not find container \"a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350\": container with ID starting with a8308c2802340a626a2efdb1a7bf1621e55499d9b818d7cb8dddf7e8eac9c350 not found: ID does not exist" Oct 30 00:25:59.155212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.155151 4433 scope.go:122] "RemoveContainer" containerID="11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7" Oct 30 00:25:59.166354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.166329 4433 scope.go:122] "RemoveContainer" containerID="11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7" Oct 30 00:25:59.166757 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:25:59.166734 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7\": container with ID starting with 11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7 not found: ID does not exist" containerID="11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7" Oct 30 00:25:59.166862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.166769 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7"} err="failed to get container status \"11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7\": rpc error: code = NotFound desc = could not find container \"11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7\": container with ID starting with 11c14b88926a7adcd7e494071da1fbfa72fecd73bdb0c856014281b1bed8ada7 not found: ID does not exist" Oct 30 00:25:59.191654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.184760 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp"] Oct 30 00:25:59.191812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.191686 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96027e-688bd56b5-hjdkp"] Oct 30 00:25:59.226919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.226846 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f"] Oct 30 00:25:59.233653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.233617 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96027e-5ffc8bd475-xhl7f"] Oct 30 00:25:59.240730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.240636 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:59.350245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.350205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:59.462296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.462138 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:59.727962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.727883 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:25:59.835435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:25:59.835400 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:00.856008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:00.855974 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:01.185069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:01.184988 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd4cf6e-0a68-44d4-959e-087bd8b443af" path="/var/lib/kubelet/pods/1cd4cf6e-0a68-44d4-959e-087bd8b443af/volumes" Oct 30 00:26:01.185770 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:01.185748 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d30c41b-626d-4e4b-8d0b-5211b589275d" path="/var/lib/kubelet/pods/2d30c41b-626d-4e4b-8d0b-5211b589275d/volumes" Oct 30 00:26:01.860516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:01.860479 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:01.968255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:01.968220 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:02.076886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.076844 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:02.138694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.138607 4433 generic.go:356] "Generic (PLEG): container finished" podID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerID="0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40" exitCode=0 Oct 30 00:26:02.138876 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.138715 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" event={"ID":"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc","Type":"ContainerDied","Data":"0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40"} Oct 30 00:26:02.139285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.139266 4433 scope.go:122] "RemoveContainer" containerID="0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40" Oct 30 00:26:02.141398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.141352 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:26:02.141529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.141457 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:26:02.141529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.141508 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:26:02.290445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.290397 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:02.905465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:02.905431 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:03.149530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:03.149488 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" event={"ID":"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc","Type":"ContainerStarted","Data":"2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd"} Oct 30 00:26:04.015832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:04.013190 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:26:04.019873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:04.019808 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:04.020159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:04.019890 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:04.539106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:04.539057 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:04.778844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:04.778777 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:04.891537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:04.891500 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:05.010167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:05.010124 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:05.010374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:05.010193 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:05.277645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:05.271989 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:05.517939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:05.517903 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:05.802473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:05.802423 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:06.010162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.010119 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:06.010376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.010203 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:06.306741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.306700 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:06.519277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.519243 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:06.626015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.625970 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:06.832384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.832350 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4"] Oct 30 00:26:06.832804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.832770 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" containerID="cri-o://599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3" gracePeriod=30 Oct 30 00:26:06.854983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:06.854952 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:07.009938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.009830 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:07.009938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.009918 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:07.119205 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:07.119157 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:26:07.333979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.333944 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:07.444169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.444132 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:07.566173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.566137 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:07.681145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.681063 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:07.802856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.802808 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:07.828715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.828678 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw"] Oct 30 00:26:07.829379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829365 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1cd4cf6e-0a68-44d4-959e-087bd8b443af" containerName="payload-pre-processing" Oct 30 00:26:07.829433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829380 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd4cf6e-0a68-44d4-959e-087bd8b443af" containerName="payload-pre-processing" Oct 30 00:26:07.829433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829392 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b2c47144-8118-490f-82ec-e2ee06a52b7c" containerName="payload-processing" Oct 30 00:26:07.829433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829398 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c47144-8118-490f-82ec-e2ee06a52b7c" containerName="payload-processing" Oct 30 00:26:07.829433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829411 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d9c76eba-4a75-4417-a6f4-21b297c54ac0" containerName="maas-api" Oct 30 00:26:07.829433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829417 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c76eba-4a75-4417-a6f4-21b297c54ac0" containerName="maas-api" Oct 30 00:26:07.829433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829423 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5ee60444-e0e5-46cb-81fc-9522041a5b22" containerName="maas-api" Oct 30 00:26:07.829433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829429 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee60444-e0e5-46cb-81fc-9522041a5b22" containerName="maas-api" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829438 4433 cpu_manager.go:414] "containerMap: removing container" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829443 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829451 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2d30c41b-626d-4e4b-8d0b-5211b589275d" containerName="maas-api" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829456 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d30c41b-626d-4e4b-8d0b-5211b589275d" containerName="maas-api" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829479 4433 cpu_manager.go:414] "containerMap: removing container" podUID="c90dfe48-38f9-463a-aecc-887fa1d2c6e0" containerName="payload-pre-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829486 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90dfe48-38f9-463a-aecc-887fa1d2c6e0" containerName="payload-pre-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829498 4433 cpu_manager.go:414] "containerMap: removing container" podUID="04e1172c-4907-4d1b-8276-fda61f96cdef" containerName="payload-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829503 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e1172c-4907-4d1b-8276-fda61f96cdef" containerName="payload-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829519 4433 cpu_manager.go:414] "containerMap: removing container" podUID="fba9c019-43cf-4e31-a9ad-1a887c2be5ff" containerName="payload-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829525 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba9c019-43cf-4e31-a9ad-1a887c2be5ff" containerName="payload-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829657 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba9c019-43cf-4e31-a9ad-1a887c2be5ff" containerName="payload-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829668 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd4cf6e-0a68-44d4-959e-087bd8b443af" containerName="payload-pre-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829676 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c47144-8118-490f-82ec-e2ee06a52b7c" containerName="payload-processing" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829684 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c76eba-4a75-4417-a6f4-21b297c54ac0" containerName="maas-api" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829691 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee60444-e0e5-46cb-81fc-9522041a5b22" containerName="maas-api" Oct 30 00:26:07.829693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829697 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c6a0e5-d146-4014-9851-0cb3734cc5f0" containerName="istio-proxy" Oct 30 00:26:07.830177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829704 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d30c41b-626d-4e4b-8d0b-5211b589275d" containerName="maas-api" Oct 30 00:26:07.830177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829713 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90dfe48-38f9-463a-aecc-887fa1d2c6e0" containerName="payload-pre-processing" Oct 30 00:26:07.830177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.829722 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e1172c-4907-4d1b-8276-fda61f96cdef" containerName="payload-processing" Oct 30 00:26:07.832774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.832751 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:07.856534 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.856498 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw"] Oct 30 00:26:07.857840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.857813 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fb3ad\"" Oct 30 00:26:07.858004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.857987 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fb3ad-dockercfg-5jpbd\"" Oct 30 00:26:07.965484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.965399 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:26:07.965713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:07.965683 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:26:08.019061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.018662 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmkgq\" (UniqueName: \"kubernetes.io/projected/d70e5015-960b-40b6-a828-b18393dfc9ad-kube-api-access-rmkgq\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:08.019061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.018784 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70e5015-960b-40b6-a828-b18393dfc9ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:08.026117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.020132 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:08.026117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.020185 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:08.120414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.120378 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmkgq\" (UniqueName: \"kubernetes.io/projected/d70e5015-960b-40b6-a828-b18393dfc9ad-kube-api-access-rmkgq\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:08.120566 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.120494 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70e5015-960b-40b6-a828-b18393dfc9ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:08.121192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.121172 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70e5015-960b-40b6-a828-b18393dfc9ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:08.144646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.144564 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmkgq\" (UniqueName: \"kubernetes.io/projected/d70e5015-960b-40b6-a828-b18393dfc9ad-kube-api-access-rmkgq\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:08.152487 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.152459 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:08.249007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.248970 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:26:08.499756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.499729 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw"] Oct 30 00:26:08.502454 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:08.502426 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70e5015_960b_40b6_a828_b18393dfc9ad.slice/crio-603de2ed9b8a99e7b998244d1b451fd79305017a4e19dd389e993720eca34fb2 WatchSource:0}: Error finding container 603de2ed9b8a99e7b998244d1b451fd79305017a4e19dd389e993720eca34fb2: Status 404 returned error can't find the container with id 603de2ed9b8a99e7b998244d1b451fd79305017a4e19dd389e993720eca34fb2 Oct 30 00:26:08.559229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:08.559188 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:09.011345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.010183 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:09.011345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.010248 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:09.205650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.205608 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" event={"ID":"d70e5015-960b-40b6-a828-b18393dfc9ad","Type":"ContainerStarted","Data":"4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b"} Oct 30 00:26:09.205865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.205671 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" event={"ID":"d70e5015-960b-40b6-a828-b18393dfc9ad","Type":"ContainerStarted","Data":"603de2ed9b8a99e7b998244d1b451fd79305017a4e19dd389e993720eca34fb2"} Oct 30 00:26:09.205956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.205864 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:09.247626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.247561 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" podStartSLOduration=2.247547111 podStartE2EDuration="2.247547111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:09.246958103 +0000 UTC m=+1278.181026638" watchObservedRunningTime="2026-10-30 00:26:09.247547111 +0000 UTC m=+1278.181615646" Oct 30 00:26:09.278295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.278215 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:09.811561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:09.797470 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:10.011169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:10.011131 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:10.011388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:10.011211 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:10.184162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:10.184066 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:10.438929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:10.438810 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:11.010055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:11.010013 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:11.010496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:11.010075 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:11.081431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:11.081393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:11.215046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:11.214677 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:11.355269 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:11.355230 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:11.471759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:11.471721 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:12.010854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.010814 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:12.011239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.010878 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:12.090937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.088412 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:12.151144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.151112 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:26:12.236827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.236791 4433 generic.go:356] "Generic (PLEG): container finished" podID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerID="599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3" exitCode=0 Oct 30 00:26:12.237019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.236859 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" event={"ID":"75ee9f6b-478a-42c9-9bb2-774948b06a17","Type":"ContainerDied","Data":"599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3"} Oct 30 00:26:12.237019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.236911 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" event={"ID":"75ee9f6b-478a-42c9-9bb2-774948b06a17","Type":"ContainerDied","Data":"9e43eff8bb010ad89a69ab0cfc197298137763ce87ef43fb45f8b4bbd9ff74c4"} Oct 30 00:26:12.237019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.236942 4433 scope.go:122] "RemoveContainer" containerID="599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3" Oct 30 00:26:12.237196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.237118 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4" Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.243985 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-kube-api-access-gcvh7\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-kube-api-access-gcvh7\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244094 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-socket\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244352 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-socket" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244427 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-token\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-token\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244455 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-envoy\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244511 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-certs\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244690 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ee9f6b-478a-42c9-9bb2-774948b06a17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee9f6b-478a-42c9-9bb2-774948b06a17-istiod-ca-cert\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244725 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-podinfo\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244756 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-credential-socket\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.244832 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-data\") pod \"75ee9f6b-478a-42c9-9bb2-774948b06a17\" (UID: \"75ee9f6b-478a-42c9-9bb2-774948b06a17\") " Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.245747 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.245949 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-data" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.246044 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-certs" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:12.246612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.246251 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ee9f6b-478a-42c9-9bb2-774948b06a17-istiod-ca-cert" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:26:12.247561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.246731 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-credential-socket" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:12.254203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.254172 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-envoy" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:12.254403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.254379 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-token" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:12.255088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.255066 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-kube-api-access-gcvh7" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "kube-api-access-gcvh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:12.255766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.255743 4433 scope.go:122] "RemoveContainer" containerID="9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c" Oct 30 00:26:12.256469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.256241 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-podinfo" pod "75ee9f6b-478a-42c9-9bb2-774948b06a17" (UID: "75ee9f6b-478a-42c9-9bb2-774948b06a17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:26:12.298630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.298606 4433 scope.go:122] "RemoveContainer" containerID="599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3" Oct 30 00:26:12.299030 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:12.299010 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3\": container with ID starting with 599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3 not found: ID does not exist" containerID="599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3" Oct 30 00:26:12.299096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.299040 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3"} err="failed to get container status \"599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3\": rpc error: code = NotFound desc = could not find container \"599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3\": container with ID starting with 599a2295e1b98398e6b3e4488b720963e7cc61cdc9a5178ab9817bafc5b24ff3 not found: ID does not exist" Oct 30 00:26:12.299096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.299063 4433 scope.go:122] "RemoveContainer" containerID="9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c" Oct 30 00:26:12.299547 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:12.299524 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c\": container with ID starting with 9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c not found: ID does not exist" containerID="9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c" Oct 30 00:26:12.299939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.299556 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c"} err="failed to get container status \"9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c\": rpc error: code = NotFound desc = could not find container \"9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c\": container with ID starting with 9138d5a0cec88532b8affbca043feada82f858b66a1491d283c41510dc7a7f5c not found: ID does not exist" Oct 30 00:26:12.347182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347148 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.347182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347178 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvh7\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-kube-api-access-gcvh7\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.347182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347189 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.347424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347198 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.347424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347206 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.347424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347214 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee9f6b-478a-42c9-9bb2-774948b06a17-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.347424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347221 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee9f6b-478a-42c9-9bb2-774948b06a17-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.347424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.347229 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee9f6b-478a-42c9-9bb2-774948b06a17-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:12.590294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.590258 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4"] Oct 30 00:26:12.603141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:12.603111 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a55e1-openshift-default-7db955fc5-52cl4"] Oct 30 00:26:13.009769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.009736 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:13.009965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.009817 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:13.182623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.182557 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" path="/var/lib/kubelet/pods/75ee9f6b-478a-42c9-9bb2-774948b06a17/volumes" Oct 30 00:26:13.623595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.622723 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:13.758824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.758781 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:13.862096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.862060 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:13.915241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.915165 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw"] Oct 30 00:26:13.915900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.915882 4433 cpu_manager.go:414] "containerMap: removing container" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" Oct 30 00:26:13.915960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.915902 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" Oct 30 00:26:13.915960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.915933 4433 cpu_manager.go:414] "containerMap: removing container" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" Oct 30 00:26:13.915960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.915939 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" Oct 30 00:26:13.916101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.916091 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" Oct 30 00:26:13.916137 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.916112 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ee9f6b-478a-42c9-9bb2-774948b06a17" containerName="istio-proxy" Oct 30 00:26:13.919382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.919360 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:13.947859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.947826 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6cfcf7\"" Oct 30 00:26:13.948076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.947900 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6cfcf7-dockercfg-gdskm\"" Oct 30 00:26:13.954522 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:13.954477 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw"] Oct 30 00:26:14.018944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.018895 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:14.019195 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.018983 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:14.110605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.110073 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmt59\" (UniqueName: \"kubernetes.io/projected/867d9d08-9c17-422a-8643-3127b4aa8589-kube-api-access-hmt59\") pod \"payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:14.110605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.110271 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867d9d08-9c17-422a-8643-3127b4aa8589-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:14.211844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.211756 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmt59\" (UniqueName: \"kubernetes.io/projected/867d9d08-9c17-422a-8643-3127b4aa8589-kube-api-access-hmt59\") pod \"payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:14.212222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.211882 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867d9d08-9c17-422a-8643-3127b4aa8589-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:14.212607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.212568 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867d9d08-9c17-422a-8643-3127b4aa8589-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:14.242034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.242005 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmt59\" (UniqueName: \"kubernetes.io/projected/867d9d08-9c17-422a-8643-3127b4aa8589-kube-api-access-hmt59\") pod \"payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:14.258501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.258309 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:14.381058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.375672 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:14.531955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.531879 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:14.767109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.767077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:14.867316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.867285 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw"] Oct 30 00:26:14.868401 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:14.868378 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867d9d08_9c17_422a_8643_3127b4aa8589.slice/crio-2a6d97a0150bd6893c7649e13ecb79be217db1cf79758334a952bceca43d313f WatchSource:0}: Error finding container 2a6d97a0150bd6893c7649e13ecb79be217db1cf79758334a952bceca43d313f: Status 404 returned error can't find the container with id 2a6d97a0150bd6893c7649e13ecb79be217db1cf79758334a952bceca43d313f Oct 30 00:26:14.885844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.885811 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:14.943787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.943758 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt"] Oct 30 00:26:14.947634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.947614 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:14.968733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.968085 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8b3bf0\"" Oct 30 00:26:14.968733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.968181 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8b3bf0-dockercfg-qqnsn\"" Oct 30 00:26:14.981789 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:14.981758 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt"] Oct 30 00:26:15.013595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.010069 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:15.013595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.010132 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:15.037188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.027768 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424d4f13-177e-4ef4-986d-60f788399800-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:15.037188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.028147 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lt72\" (UniqueName: \"kubernetes.io/projected/424d4f13-177e-4ef4-986d-60f788399800-kube-api-access-4lt72\") pod \"payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:15.099370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.049031 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x"] Oct 30 00:26:15.099370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.054114 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.099370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.098194 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x"] Oct 30 00:26:15.133651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.130195 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580149e6-952f-43dc-84cd-4e004513bbbc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.133651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.130408 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lt72\" (UniqueName: \"kubernetes.io/projected/424d4f13-177e-4ef4-986d-60f788399800-kube-api-access-4lt72\") pod \"payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:15.133651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.130478 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4q2\" (UniqueName: \"kubernetes.io/projected/580149e6-952f-43dc-84cd-4e004513bbbc-kube-api-access-wf4q2\") pod \"payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.133651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.130538 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424d4f13-177e-4ef4-986d-60f788399800-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:15.133651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.131668 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424d4f13-177e-4ef4-986d-60f788399800-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:15.164080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.164040 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lt72\" (UniqueName: \"kubernetes.io/projected/424d4f13-177e-4ef4-986d-60f788399800-kube-api-access-4lt72\") pod \"payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:15.231925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.231883 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4q2\" (UniqueName: \"kubernetes.io/projected/580149e6-952f-43dc-84cd-4e004513bbbc-kube-api-access-wf4q2\") pod \"payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.232422 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.232080 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580149e6-952f-43dc-84cd-4e004513bbbc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.232968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.232939 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580149e6-952f-43dc-84cd-4e004513bbbc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.256840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.256320 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4q2\" (UniqueName: \"kubernetes.io/projected/580149e6-952f-43dc-84cd-4e004513bbbc-kube-api-access-wf4q2\") pod \"payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.262879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.262851 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:15.281814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.281767 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" event={"ID":"867d9d08-9c17-422a-8643-3127b4aa8589","Type":"ContainerStarted","Data":"19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2"} Oct 30 00:26:15.281983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.281836 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" event={"ID":"867d9d08-9c17-422a-8643-3127b4aa8589","Type":"ContainerStarted","Data":"2a6d97a0150bd6893c7649e13ecb79be217db1cf79758334a952bceca43d313f"} Oct 30 00:26:15.283082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.283056 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:15.325832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.325771 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" podStartSLOduration=2.325754449 podStartE2EDuration="2.325754449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:15.323445318 +0000 UTC m=+1284.257513878" watchObservedRunningTime="2026-10-30 00:26:15.325754449 +0000 UTC m=+1284.259822985" Oct 30 00:26:15.401117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.389846 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:15.634364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.634331 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt"] Oct 30 00:26:15.674526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.674492 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:15.791136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:15.786115 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x"] Oct 30 00:26:15.791629 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:15.791585 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580149e6_952f_43dc_84cd_4e004513bbbc.slice/crio-5c7e2d3df186710f4cd7ab3443395f3b59cfef0d23a2ac29aaad9d55b4cd77fa WatchSource:0}: Error finding container 5c7e2d3df186710f4cd7ab3443395f3b59cfef0d23a2ac29aaad9d55b4cd77fa: Status 404 returned error can't find the container with id 5c7e2d3df186710f4cd7ab3443395f3b59cfef0d23a2ac29aaad9d55b4cd77fa Oct 30 00:26:16.010180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.010088 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:16.010180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.010148 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:16.291155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.291044 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" event={"ID":"580149e6-952f-43dc-84cd-4e004513bbbc","Type":"ContainerStarted","Data":"70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7"} Oct 30 00:26:16.291155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.291123 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:16.291155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.291147 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" event={"ID":"580149e6-952f-43dc-84cd-4e004513bbbc","Type":"ContainerStarted","Data":"5c7e2d3df186710f4cd7ab3443395f3b59cfef0d23a2ac29aaad9d55b4cd77fa"} Oct 30 00:26:16.293054 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.293022 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" event={"ID":"424d4f13-177e-4ef4-986d-60f788399800","Type":"ContainerStarted","Data":"fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d"} Oct 30 00:26:16.293172 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.293071 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" event={"ID":"424d4f13-177e-4ef4-986d-60f788399800","Type":"ContainerStarted","Data":"1dbb70fefd4d1f8570f9f125c246b23bfdb0273c169bc84ff98d06eef7a3f6bd"} Oct 30 00:26:16.293227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.293194 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:16.335332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.335279 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" podStartSLOduration=1.3352649730000001 podStartE2EDuration="1.335264973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:16.331499093 +0000 UTC m=+1285.265567629" watchObservedRunningTime="2026-10-30 00:26:16.335264973 +0000 UTC m=+1285.269333507" Oct 30 00:26:16.379050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.378991 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" podStartSLOduration=2.378976424 podStartE2EDuration="2.378976424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:16.372691699 +0000 UTC m=+1285.306760251" watchObservedRunningTime="2026-10-30 00:26:16.378976424 +0000 UTC m=+1285.313044958" Oct 30 00:26:16.628298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.628252 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:16.731441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:16.731408 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:17.011371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:17.011279 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:17.011371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:17.011335 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:17.454808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:17.454771 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:17.566176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:17.566136 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:17.668671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:17.668637 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:18.011011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:18.010969 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:18.011189 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:18.011034 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:18.431967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:18.431927 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:18.898807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:18.898768 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:19.005013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:19.004977 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:19.010723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:19.010685 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:19.010918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:19.010761 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:19.118084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:19.117863 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:19.682867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:19.682800 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:20.010258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:20.010163 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:20.010258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:20.010234 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:20.220729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:20.220693 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:26:20.409894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:20.409861 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t"] Oct 30 00:26:20.410098 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:20.410054 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" containerID="cri-o://85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5" gracePeriod=30 Oct 30 00:26:20.544317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:20.544286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:20.981633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:20.981557 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:21.009964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:21.009932 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:21.010138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:21.009984 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:22.009852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:22.009817 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:22.010270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:22.009873 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:22.388983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:22.388942 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:22.979995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:22.979956 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:23.009674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.009630 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:23.009844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.009712 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:23.164915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.164854 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:23.269626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.269505 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:23.444492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.444448 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz"] Oct 30 00:26:23.449534 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.449510 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.458457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.458427 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz"] Oct 30 00:26:23.472038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.471991 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f649b0-openshift-default-dockercfg-cvkf8\"" Oct 30 00:26:23.488384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.488352 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:23.594233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594190 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-podinfo\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594408 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-credential-socket\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594465 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-token\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594529 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-certs\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594560 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istiod-ca-cert\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594677 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-socket\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594713 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmxk\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-kube-api-access-gzmxk\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594747 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-envoy\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.594993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.594782 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-data\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.696454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.696404 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-socket\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.696706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.696474 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmxk\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-kube-api-access-gzmxk\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.696706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.696510 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-envoy\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.696706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.696546 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-data\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.696706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.696660 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-podinfo\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.696948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.696876 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-credential-socket\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.696948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.696896 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-socket\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.697201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.697180 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-credential-socket\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.697284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.697225 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-token\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.697284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.697227 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-data\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.697387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.697316 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-certs\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.697387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.697352 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istiod-ca-cert\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.697770 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.697720 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-certs\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.698170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.698148 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istiod-ca-cert\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.699945 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.699922 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-podinfo\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.703277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.703247 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-envoy\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.722641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.722548 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmxk\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-kube-api-access-gzmxk\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.722897 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.722846 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-token\") pod \"e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:23.751358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.751325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:23.774989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:23.774937 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:24.010663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.010623 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:24.010894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.010704 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:24.115423 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:24.115396 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5c34ff_6315_43bd_af4e_67ca3bb5caa3.slice/crio-c74853cc46a3c7bfe6f40b397328f8a0d88a8c322ceb97ad79823dd6d8adb08d WatchSource:0}: Error finding container c74853cc46a3c7bfe6f40b397328f8a0d88a8c322ceb97ad79823dd6d8adb08d: Status 404 returned error can't find the container with id c74853cc46a3c7bfe6f40b397328f8a0d88a8c322ceb97ad79823dd6d8adb08d Oct 30 00:26:24.116075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.116047 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz"] Oct 30 00:26:24.118047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.118014 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:26:24.118142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.118079 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:26:24.118142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.118108 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:26:24.288999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.288967 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:24.368831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.368743 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" event={"ID":"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3","Type":"ContainerStarted","Data":"a78292dcc1fa741ea73e20651f88982aee80a5b446108df76aac0c5ea77ab098"} Oct 30 00:26:24.368831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.368801 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" event={"ID":"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3","Type":"ContainerStarted","Data":"c74853cc46a3c7bfe6f40b397328f8a0d88a8c322ceb97ad79823dd6d8adb08d"} Oct 30 00:26:24.416845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.416785 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podStartSLOduration=1.416767155 podStartE2EDuration="1.416767155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:24.414455518 +0000 UTC m=+1293.348524054" watchObservedRunningTime="2026-10-30 00:26:24.416767155 +0000 UTC m=+1293.350835725" Oct 30 00:26:24.775961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.775649 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:24.839064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.839032 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:24.942638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:24.942591 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:25.010375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.010339 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:25.010553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.010395 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:25.724561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.724533 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:26:25.767670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.767635 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-kube-api-access-fmgjz\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-kube-api-access-fmgjz\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.767861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.767798 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3493edb6-a29a-43ce-a229-5dde741d4733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3493edb6-a29a-43ce-a229-5dde741d4733-istiod-ca-cert\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.767861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.767838 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3493edb6-a29a-43ce-a229-5dde741d4733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3493edb6-a29a-43ce-a229-5dde741d4733-istio-podinfo\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.767997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.767979 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-credential-socket\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.768051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768016 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-istio-token\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-istio-token\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.768051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768039 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-socket\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.768167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768091 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-certs\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.768225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768211 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-data\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.768279 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768242 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-envoy\") pod \"3493edb6-a29a-43ce-a229-5dde741d4733\" (UID: \"3493edb6-a29a-43ce-a229-5dde741d4733\") " Oct 30 00:26:25.768336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768286 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-credential-socket" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:25.768396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768328 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3493edb6-a29a-43ce-a229-5dde741d4733-istiod-ca-cert" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:26:25.768396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768340 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-socket" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:25.768497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768426 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-certs" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:25.768558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768498 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-data" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:25.768961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768935 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.768961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768959 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.769145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768969 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.769145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768977 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.769145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.768984 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3493edb6-a29a-43ce-a229-5dde741d4733-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.770350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.770325 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-envoy" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:25.770441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.770332 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-istio-token" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:25.770544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.770526 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3493edb6-a29a-43ce-a229-5dde741d4733-istio-podinfo" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:26:25.770621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.770548 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-kube-api-access-fmgjz" pod "3493edb6-a29a-43ce-a229-5dde741d4733" (UID: "3493edb6-a29a-43ce-a229-5dde741d4733"). InnerVolumeSpecName "kube-api-access-fmgjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:25.778144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.778118 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:26:25.778304 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.778187 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 00:26:25.870552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.870517 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmgjz\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-kube-api-access-fmgjz\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.870552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.870548 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3493edb6-a29a-43ce-a229-5dde741d4733-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.870552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.870557 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3493edb6-a29a-43ce-a229-5dde741d4733-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:25.870774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:25.870565 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3493edb6-a29a-43ce-a229-5dde741d4733-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:26.009797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.009709 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:26.009797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.009776 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:26.333703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.333670 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:26.384775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.384736 4433 generic.go:356] "Generic (PLEG): container finished" podID="3493edb6-a29a-43ce-a229-5dde741d4733" containerID="85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5" exitCode=0 Oct 30 00:26:26.384973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.384787 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" Oct 30 00:26:26.384973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.384815 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" event={"ID":"3493edb6-a29a-43ce-a229-5dde741d4733","Type":"ContainerDied","Data":"85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5"} Oct 30 00:26:26.384973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.384869 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t" event={"ID":"3493edb6-a29a-43ce-a229-5dde741d4733","Type":"ContainerDied","Data":"1ccb61e7a32500a691a62b832e4427552be5fbebcc1c221f3a14722d3a07f4d0"} Oct 30 00:26:26.384973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.384908 4433 scope.go:122] "RemoveContainer" containerID="85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5" Oct 30 00:26:26.398358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.398336 4433 scope.go:122] "RemoveContainer" containerID="85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5" Oct 30 00:26:26.398803 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:26.398759 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5\": container with ID starting with 85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5 not found: ID does not exist" containerID="85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5" Oct 30 00:26:26.398980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.398799 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5"} err="failed to get container status \"85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5\": rpc error: code = NotFound desc = could not find container \"85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5\": container with ID starting with 85ba84a497e49132d67c94286a9351f0ed34361b791ad5b7f078d76d37316ad5 not found: ID does not exist" Oct 30 00:26:26.441626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.441588 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t"] Oct 30 00:26:26.454169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.454134 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-d469t"] Oct 30 00:26:26.731954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.731854 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:26.780422 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.780379 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:26.840485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:26.840451 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:27.010030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.009938 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:27.010030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.010015 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:27.181801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.181766 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" path="/var/lib/kubelet/pods/3493edb6-a29a-43ce-a229-5dde741d4733/volumes" Oct 30 00:26:27.307994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.307966 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:26:27.308165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.308054 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:26:27.308165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.308097 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:26:27.414003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.413965 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:27.415405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.415380 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:26:27.903103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.903068 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5"] Oct 30 00:26:27.903833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.903816 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" Oct 30 00:26:27.903833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.903833 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" Oct 30 00:26:27.903967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.903957 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3493edb6-a29a-43ce-a229-5dde741d4733" containerName="istio-proxy" Oct 30 00:26:27.907334 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.907313 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:27.928082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.928049 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6cfcf7-serving-cert\"" Oct 30 00:26:27.942650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.942609 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5"] Oct 30 00:26:27.990248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:27.990212 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:28.011408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.011354 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:28.011564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.011431 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:28.013523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.013490 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sq42\" (UniqueName: \"kubernetes.io/projected/bccd33e5-4ecd-4528-b879-a460841f4e90-kube-api-access-8sq42\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.013767 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.013747 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-maas-api-tls\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.013848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.013794 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccd33e5-4ecd-4528-b879-a460841f4e90-openshift-service-ca\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.013897 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.013872 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-metrics-certs\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.099252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.099214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:28.116055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.116020 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-maas-api-tls\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.116239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.116083 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccd33e5-4ecd-4528-b879-a460841f4e90-openshift-service-ca\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.116239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.116120 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-metrics-certs\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.116438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.116413 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sq42\" (UniqueName: \"kubernetes.io/projected/bccd33e5-4ecd-4528-b879-a460841f4e90-kube-api-access-8sq42\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.116855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.116827 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccd33e5-4ecd-4528-b879-a460841f4e90-openshift-service-ca\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.118652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.118632 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-metrics-certs\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.118737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.118677 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-maas-api-tls\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.141260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.141222 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sq42\" (UniqueName: \"kubernetes.io/projected/bccd33e5-4ecd-4528-b879-a460841f4e90-kube-api-access-8sq42\") pod \"maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.204718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.204608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:28.222712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.222669 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:28.586511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:28.586477 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5"] Oct 30 00:26:28.587329 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:28.587298 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccd33e5_4ecd_4528_b879_a460841f4e90.slice/crio-612074ff3f5e22606cb2aa8342a745cdf71e2a88f81ed3d7a0f217d5c0d078ab WatchSource:0}: Error finding container 612074ff3f5e22606cb2aa8342a745cdf71e2a88f81ed3d7a0f217d5c0d078ab: Status 404 returned error can't find the container with id 612074ff3f5e22606cb2aa8342a745cdf71e2a88f81ed3d7a0f217d5c0d078ab Oct 30 00:26:29.010692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.010658 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:29.011149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.010729 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:29.094105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.094063 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:29.204186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.203921 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:29.670592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.655682 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:29.670592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.668740 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" event={"ID":"bccd33e5-4ecd-4528-b879-a460841f4e90","Type":"ContainerStarted","Data":"f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d"} Oct 30 00:26:29.670592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.668802 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" event={"ID":"bccd33e5-4ecd-4528-b879-a460841f4e90","Type":"ContainerStarted","Data":"612074ff3f5e22606cb2aa8342a745cdf71e2a88f81ed3d7a0f217d5c0d078ab"} Oct 30 00:26:29.670592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.668862 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:29.740351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.736077 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" podStartSLOduration=2.4211363439999998 podStartE2EDuration="2.736061772s" totalImagesPullingTime="314.925428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:29.73391007 +0000 UTC m=+1298.667978606" watchObservedRunningTime="2026-10-30 00:26:29.736061772 +0000 UTC m=+1298.670130306" Oct 30 00:26:29.991709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.991625 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb"] Oct 30 00:26:29.995467 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:29.995447 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.011454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.011397 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:30.011875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.011482 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:30.013661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.013637 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8b3bf0-serving-cert\"" Oct 30 00:26:30.022474 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.022438 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb"] Oct 30 00:26:30.029759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.029722 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlnk\" (UniqueName: \"kubernetes.io/projected/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-kube-api-access-qmlnk\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.029930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.029895 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.030113 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.030088 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.030264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.030244 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-metrics-certs\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.082599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.082525 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:30.132212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.132173 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlnk\" (UniqueName: \"kubernetes.io/projected/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-kube-api-access-qmlnk\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.132393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.132274 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.132444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.132417 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.132558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.132542 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-metrics-certs\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.132645 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:30.132586 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-serving-cert: secret "maas-api-e2e-praxis-a-8b3bf0-serving-cert" not found Oct 30 00:26:30.132691 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:30.132666 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls podName:e0cd303d-ba4a-4dd1-a4fa-442846dff3d7 nodeName:}" failed. No retries permitted until 2026-10-30 00:26:30.632641613 +0000 UTC m=+1299.566710133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls") pod "maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" (UID: "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7") : secret "maas-api-e2e-praxis-a-8b3bf0-serving-cert" not found Oct 30 00:26:30.133000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.132976 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.135266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.135237 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-metrics-certs\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.201032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.197349 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlnk\" (UniqueName: \"kubernetes.io/projected/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-kube-api-access-qmlnk\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.208913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.208875 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:30.316687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.316647 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:30.459599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.455888 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:30.619650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.618203 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:30.712591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.704391 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.724462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.724405 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:30.909423 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:30.909324 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:31.016600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.011978 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:31.016600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.012043 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:31.288604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.288401 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb"] Oct 30 00:26:31.289134 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:31.289099 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cd303d_ba4a_4dd1_a4fa_442846dff3d7.slice/crio-85caefca5805365916d373e4cf649702c1ba69ca971fe17c5d982e9a7496694c WatchSource:0}: Error finding container 85caefca5805365916d373e4cf649702c1ba69ca971fe17c5d982e9a7496694c: Status 404 returned error can't find the container with id 85caefca5805365916d373e4cf649702c1ba69ca971fe17c5d982e9a7496694c Oct 30 00:26:31.740430 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.740391 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" event={"ID":"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7","Type":"ContainerStarted","Data":"4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401"} Oct 30 00:26:31.740638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.740480 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" event={"ID":"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7","Type":"ContainerStarted","Data":"85caefca5805365916d373e4cf649702c1ba69ca971fe17c5d982e9a7496694c"} Oct 30 00:26:31.740638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.740613 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:31.796161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.796122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:31.796733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:31.796691 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" podStartSLOduration=2.4846489529999998 podStartE2EDuration="2.796677358s" totalImagesPullingTime="312.028405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:31.792443208 +0000 UTC m=+1300.726511746" watchObservedRunningTime="2026-10-30 00:26:31.796677358 +0000 UTC m=+1300.730745891" Oct 30 00:26:32.009908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:32.009816 4433 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:26:32.009908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:32.009888 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 00:26:33.114833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.114798 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc"] Oct 30 00:26:33.118328 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.118305 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.141778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.141560 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:26:33.143036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.143009 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-bb16a8-serving-cert\"" Oct 30 00:26:33.143856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.143833 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:26:33.151725 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.151663 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc"] Oct 30 00:26:33.154705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.153083 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-metrics-certs\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.154705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.153229 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgcr\" (UniqueName: \"kubernetes.io/projected/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-kube-api-access-jvgcr\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.154705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.153774 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-maas-api-tls\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.154705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.153810 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.199900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.199785 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:33.255557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.255516 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-maas-api-tls\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.255557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.255565 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.255837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.255635 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-metrics-certs\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.255837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.255781 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgcr\" (UniqueName: \"kubernetes.io/projected/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-kube-api-access-jvgcr\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.256981 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.256952 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.260763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.260735 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-maas-api-tls\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.263106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.263085 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-metrics-certs\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.280432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.280404 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgcr\" (UniqueName: \"kubernetes.io/projected/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-kube-api-access-jvgcr\") pod \"maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.301362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.301312 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:33.438735 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.438646 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:33.755816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:33.755785 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc"] Oct 30 00:26:33.756305 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:33.756282 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b98754c_b660_4dbe_aa39_a01a1fb1fe16.slice/crio-656e7dcb5d356e35ccc854793c3538305a3ed12191c9c2f6e3dcc9f0c0e79399 WatchSource:0}: Error finding container 656e7dcb5d356e35ccc854793c3538305a3ed12191c9c2f6e3dcc9f0c0e79399: Status 404 returned error can't find the container with id 656e7dcb5d356e35ccc854793c3538305a3ed12191c9c2f6e3dcc9f0c0e79399 Oct 30 00:26:34.773010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:34.772923 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" event={"ID":"1b98754c-b660-4dbe-aa39-a01a1fb1fe16","Type":"ContainerStarted","Data":"18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b"} Oct 30 00:26:34.773010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:34.772982 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" event={"ID":"1b98754c-b660-4dbe-aa39-a01a1fb1fe16","Type":"ContainerStarted","Data":"656e7dcb5d356e35ccc854793c3538305a3ed12191c9c2f6e3dcc9f0c0e79399"} Oct 30 00:26:34.773562 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:34.773027 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:34.780217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:34.780189 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:34.823294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:34.823236 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" podStartSLOduration=1.469490522 podStartE2EDuration="1.823218979s" totalImagesPullingTime="353.728457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:34.818896384 +0000 UTC m=+1303.752964919" watchObservedRunningTime="2026-10-30 00:26:34.823218979 +0000 UTC m=+1303.757287518" Oct 30 00:26:34.890011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:34.889960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:35.735731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:35.735687 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:26:36.275536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.275495 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f"] Oct 30 00:26:36.276077 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.275817 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" containerID="cri-o://702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef" gracePeriod=30 Oct 30 00:26:36.364835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.364803 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp"] Oct 30 00:26:36.368651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.368629 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.391739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.391709 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-522ea7-serving-cert\"" Oct 30 00:26:36.410173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.410144 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp"] Oct 30 00:26:36.524187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.524154 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:36.528413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.528356 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-metrics-certs\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.528535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.528420 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51c7cd0-80c4-48cb-a6be-67054065240c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.528535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.528485 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsc6r\" (UniqueName: \"kubernetes.io/projected/d51c7cd0-80c4-48cb-a6be-67054065240c-kube-api-access-jsc6r\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.528535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.528515 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.629810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.629777 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-metrics-certs\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.630036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.629852 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51c7cd0-80c4-48cb-a6be-67054065240c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.630036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.629873 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsc6r\" (UniqueName: \"kubernetes.io/projected/d51c7cd0-80c4-48cb-a6be-67054065240c-kube-api-access-jsc6r\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.630036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.629894 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.630512 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.630474 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:36.630758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.630736 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51c7cd0-80c4-48cb-a6be-67054065240c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.632452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.632428 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-metrics-certs\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.632608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.632588 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.656956 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.656927 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsc6r\" (UniqueName: \"kubernetes.io/projected/d51c7cd0-80c4-48cb-a6be-67054065240c-kube-api-access-jsc6r\") pod \"maas-api-e2e-praxis-a-522ea7-586d998585-6smkp\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:36.685680 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:36.685652 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:37.114858 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:37.114816 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:26:37.130960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:37.130933 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp"] Oct 30 00:26:37.132014 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:37.131984 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51c7cd0_80c4_48cb_a6be_67054065240c.slice/crio-30ec9bdc0d7e95d18cd7115109374307ef47ebacaa4029d82c5a6e82c367e049 WatchSource:0}: Error finding container 30ec9bdc0d7e95d18cd7115109374307ef47ebacaa4029d82c5a6e82c367e049: Status 404 returned error can't find the container with id 30ec9bdc0d7e95d18cd7115109374307ef47ebacaa4029d82c5a6e82c367e049 Oct 30 00:26:37.754060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:37.753983 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:26:37.853414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:37.852753 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" event={"ID":"d51c7cd0-80c4-48cb-a6be-67054065240c","Type":"ContainerStarted","Data":"3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1"} Oct 30 00:26:37.853414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:37.852854 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" event={"ID":"d51c7cd0-80c4-48cb-a6be-67054065240c","Type":"ContainerStarted","Data":"30ec9bdc0d7e95d18cd7115109374307ef47ebacaa4029d82c5a6e82c367e049"} Oct 30 00:26:37.854440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:37.854416 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:37.902107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:37.902056 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" podStartSLOduration=1.599543874 podStartE2EDuration="1.902042242s" totalImagesPullingTime="302.498368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:37.897259387 +0000 UTC m=+1306.831327923" watchObservedRunningTime="2026-10-30 00:26:37.902042242 +0000 UTC m=+1306.836110776" Oct 30 00:26:38.166860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:38.166824 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:38.303710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:38.303667 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:39.759268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:39.759235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:40.788690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:40.788659 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:26:41.159394 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.159355 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:41.262045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.262013 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:41.572935 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.572908 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:26:41.751134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751011 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-data\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751065 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-credential-socket\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751112 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-kube-api-access-zzfr4\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-kube-api-access-zzfr4\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751181 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-podinfo\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751233 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-envoy\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751367 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istiod-ca-cert\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751401 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-token\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-token\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751650 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-socket\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.751766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.751702 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-certs\") pod \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\" (UID: \"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f\") " Oct 30 00:26:41.752540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.752512 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istiod-ca-cert" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:26:41.752767 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.752737 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-credential-socket" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:41.752864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.752789 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-data" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:41.752864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.752806 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-certs" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:41.753014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.752966 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-socket" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:41.755028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.754954 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-podinfo" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:26:41.755028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.754981 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-envoy" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:41.755028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.755016 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-token" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:41.755205 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.755063 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-kube-api-access-zzfr4" pod "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" (UID: "532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f"). InnerVolumeSpecName "kube-api-access-zzfr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:41.854322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854281 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854315 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854327 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzfr4\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-kube-api-access-zzfr4\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854338 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854347 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854354 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854362 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854371 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.854832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.854379 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:26:41.907367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.907332 4433 generic.go:356] "Generic (PLEG): container finished" podID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerID="702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef" exitCode=0 Oct 30 00:26:41.907536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.907392 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" Oct 30 00:26:41.907536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.907418 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" event={"ID":"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f","Type":"ContainerDied","Data":"702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef"} Oct 30 00:26:41.907536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.907513 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f" event={"ID":"532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f","Type":"ContainerDied","Data":"61d856ba1121f488e5465ae332bcae3bb68c6092a5f6871cf062a0cb2e9ca08c"} Oct 30 00:26:41.907765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.907542 4433 scope.go:122] "RemoveContainer" containerID="702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef" Oct 30 00:26:41.919320 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.919300 4433 scope.go:122] "RemoveContainer" containerID="702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef" Oct 30 00:26:41.919621 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:26:41.919599 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef\": container with ID starting with 702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef not found: ID does not exist" containerID="702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef" Oct 30 00:26:41.919673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.919633 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef"} err="failed to get container status \"702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef\": rpc error: code = NotFound desc = could not find container \"702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef\": container with ID starting with 702e6d1193f1db1eec5c6af4f2757b87f77d222f0c097fcc2fc12b9bc054d2ef not found: ID does not exist" Oct 30 00:26:41.964032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.963997 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f"] Oct 30 00:26:41.969258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:41.969229 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-mdz8f"] Oct 30 00:26:42.550052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:42.550007 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:43.181610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:43.181560 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" path="/var/lib/kubelet/pods/532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f/volumes" Oct 30 00:26:44.482645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:44.482599 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:44.891405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:44.891377 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:26:46.111547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:46.111498 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:46.315195 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:46.315141 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:47.592065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:47.592027 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:49.056004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:49.055968 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:50.894238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.894201 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk"] Oct 30 00:26:50.895232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.895208 4433 cpu_manager.go:414] "containerMap: removing container" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" Oct 30 00:26:50.895351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.895233 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" Oct 30 00:26:50.895520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.895504 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="532bb5f8-3ce8-46e7-8dd9-e8e5cf03771f" containerName="istio-proxy" Oct 30 00:26:50.899177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.899153 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:50.923101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.918714 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f649b0-serving-cert\"" Oct 30 00:26:50.925070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.925043 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk"] Oct 30 00:26:50.962455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:50.962422 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:51.018088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.018054 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.018264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.018120 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.018306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.018285 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.018362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.018350 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdc9\" (UniqueName: \"kubernetes.io/projected/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-kube-api-access-gpdc9\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.061684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.061654 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:51.120419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.120385 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.120607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.120440 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.120607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.120546 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.120694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.120615 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdc9\" (UniqueName: \"kubernetes.io/projected/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-kube-api-access-gpdc9\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.121293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.121264 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.123016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.122995 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.123099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.123041 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.160472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.160393 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdc9\" (UniqueName: \"kubernetes.io/projected/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-kube-api-access-gpdc9\") pod \"maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.166392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.166301 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:51.215733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.215704 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:51.534799 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:26:51.534759 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fa4b2c_7828_49b8_8a33_cec310f66d3d.slice/crio-0e0e2c04d9cefd11d1412d6ed4f3af17905a5aa972fb9104a6ec7c04f1008868 WatchSource:0}: Error finding container 0e0e2c04d9cefd11d1412d6ed4f3af17905a5aa972fb9104a6ec7c04f1008868: Status 404 returned error can't find the container with id 0e0e2c04d9cefd11d1412d6ed4f3af17905a5aa972fb9104a6ec7c04f1008868 Oct 30 00:26:51.534939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.534868 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk"] Oct 30 00:26:51.993166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.992812 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" event={"ID":"c4fa4b2c-7828-49b8-8a33-cec310f66d3d","Type":"ContainerStarted","Data":"4ff1e4ba32c523d0b8e8f4b7b98728263201c3dcae193c95cd19cea7d0cab6ee"} Oct 30 00:26:51.993166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.992881 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" event={"ID":"c4fa4b2c-7828-49b8-8a33-cec310f66d3d","Type":"ContainerStarted","Data":"0e0e2c04d9cefd11d1412d6ed4f3af17905a5aa972fb9104a6ec7c04f1008868"} Oct 30 00:26:51.994141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:51.994117 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:26:52.043486 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:52.043434 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" podStartSLOduration=1.734570427 podStartE2EDuration="2.04341961s" totalImagesPullingTime="308.849183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:52.041910649 +0000 UTC m=+1320.975979210" watchObservedRunningTime="2026-10-30 00:26:52.04341961 +0000 UTC m=+1320.977488144" Oct 30 00:26:52.567462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:52.567430 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:54.073269 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:54.073228 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:55.524651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:55.524602 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:56.854807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:56.854771 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:57.091276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:57.091238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:57.641635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:57.641597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:58.096179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:58.096142 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:58.438170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:58.438076 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:58.855925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:58.855888 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:26:59.018957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:26:59.018926 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:27:00.320792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:00.320760 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:00.432938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:00.432905 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:00.533546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:00.533513 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:01.971100 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:01.971062 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:02.182440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:02.182400 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:02.289421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:02.289274 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:02.422533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:02.422500 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:03.474261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:03.474225 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:03.727556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:03.727471 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:04.000067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:03.999993 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:04.119996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:04.119961 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:04.229375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:04.229325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:04.330200 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:04.330149 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:05.373173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:05.373127 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:05.488271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:05.488226 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:06.387617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:06.387563 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:06.501488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:06.501452 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:07.100523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:07.100485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:07.101240 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:27:07.101214 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:27:08.708065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:08.708033 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:08.824029 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:08.823997 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:08.933387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:08.933355 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:10.091824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:10.091787 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:10.407463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:10.407369 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:10.516039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:10.516006 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:10.634840 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:10.634807 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:10.748327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:10.748242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:10.862014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:10.861977 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:11.362170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:11.362129 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:11.649915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:11.649834 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47"] Oct 30 00:27:11.650101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:11.650059 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" containerID="cri-o://436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382" gracePeriod=30 Oct 30 00:27:11.871297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:11.871260 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:11.989277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:11.989115 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:12.341865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.341674 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:12.445181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.445140 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:12.513203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.513161 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x"] Oct 30 00:27:12.522594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.518877 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.537000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.536422 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-20d942\"" Oct 30 00:27:12.537000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.536563 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-20d942-dockercfg-d2kg9\"" Oct 30 00:27:12.572597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.560650 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9e143b-bb92-4c5d-bcb1-470341b19946-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.572597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.560818 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbsl\" (UniqueName: \"kubernetes.io/projected/9a9e143b-bb92-4c5d-bcb1-470341b19946-kube-api-access-lpbsl\") pod \"payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.572597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.572356 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x"] Oct 30 00:27:12.664698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.664024 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9e143b-bb92-4c5d-bcb1-470341b19946-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.664698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.664203 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbsl\" (UniqueName: \"kubernetes.io/projected/9a9e143b-bb92-4c5d-bcb1-470341b19946-kube-api-access-lpbsl\") pod \"payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.666634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.665196 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9e143b-bb92-4c5d-bcb1-470341b19946-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.696409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.696373 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbsl\" (UniqueName: \"kubernetes.io/projected/9a9e143b-bb92-4c5d-bcb1-470341b19946-kube-api-access-lpbsl\") pod \"payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.845632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.845596 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:12.884591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:12.880894 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:13.188442 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:13.188411 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x"] Oct 30 00:27:13.189166 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:27:13.189127 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9e143b_bb92_4c5d_bcb1_470341b19946.slice/crio-bdf25b689c49b0b3163159a5a0cb62308594d70dc7c556de62e0710fb8305dd7 WatchSource:0}: Error finding container bdf25b689c49b0b3163159a5a0cb62308594d70dc7c556de62e0710fb8305dd7: Status 404 returned error can't find the container with id bdf25b689c49b0b3163159a5a0cb62308594d70dc7c556de62e0710fb8305dd7 Oct 30 00:27:14.175322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:14.175273 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" event={"ID":"9a9e143b-bb92-4c5d-bcb1-470341b19946","Type":"ContainerStarted","Data":"79b209d09bf9c13f39ff234c5b8df01dbaaeeac8063cbddf09296862737eddb8"} Oct 30 00:27:14.175781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:14.175467 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:14.175781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:14.175487 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" event={"ID":"9a9e143b-bb92-4c5d-bcb1-470341b19946","Type":"ContainerStarted","Data":"bdf25b689c49b0b3163159a5a0cb62308594d70dc7c556de62e0710fb8305dd7"} Oct 30 00:27:14.219965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:14.219915 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" podStartSLOduration=2.219901942 podStartE2EDuration="2.219901942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:14.21610288 +0000 UTC m=+1343.150171414" watchObservedRunningTime="2026-10-30 00:27:14.219901942 +0000 UTC m=+1343.153970476" Oct 30 00:27:14.382854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:14.382798 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:15.979339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:15.979298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:16.179196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:16.179162 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:16.302415 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:16.302199 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:16.939407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:16.939383 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:27:17.069250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069215 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91ac229-73ad-4732-b919-9d35d165227c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91ac229-73ad-4732-b919-9d35d165227c-istiod-ca-cert\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.069754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069282 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-kube-api-access-k8wcv\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-kube-api-access-k8wcv\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.069754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069311 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-data\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.069754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069362 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-socket\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.069754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069470 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-credential-socket\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.069754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069655 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-istio-token\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-istio-token\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.069754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069708 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-envoy\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.070178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069770 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-socket" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:17.070178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069809 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-data" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:17.070178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069779 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91ac229-73ad-4732-b919-9d35d165227c-istiod-ca-cert" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:17.070178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069823 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a91ac229-73ad-4732-b919-9d35d165227c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91ac229-73ad-4732-b919-9d35d165227c-istio-podinfo\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.070178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069827 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-credential-socket" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:17.070178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.069881 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-certs\") pod \"a91ac229-73ad-4732-b919-9d35d165227c\" (UID: \"a91ac229-73ad-4732-b919-9d35d165227c\") " Oct 30 00:27:17.070178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.070158 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-certs" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:17.070727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.070706 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91ac229-73ad-4732-b919-9d35d165227c-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.070784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.070730 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.070784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.070739 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.070784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.070752 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.070784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.070761 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.072071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.072035 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-envoy" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:17.072144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.072093 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-kube-api-access-k8wcv" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "kube-api-access-k8wcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:17.072363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.072347 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-istio-token" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:17.072363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.072349 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a91ac229-73ad-4732-b919-9d35d165227c-istio-podinfo" pod "a91ac229-73ad-4732-b919-9d35d165227c" (UID: "a91ac229-73ad-4732-b919-9d35d165227c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:17.172235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.172159 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.172235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.172188 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91ac229-73ad-4732-b919-9d35d165227c-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.172235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.172198 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91ac229-73ad-4732-b919-9d35d165227c-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.172235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.172207 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8wcv\" (UniqueName: \"kubernetes.io/projected/a91ac229-73ad-4732-b919-9d35d165227c-kube-api-access-k8wcv\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:27:17.201395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.201356 4433 generic.go:356] "Generic (PLEG): container finished" podID="a91ac229-73ad-4732-b919-9d35d165227c" containerID="436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382" exitCode=0 Oct 30 00:27:17.201549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.201396 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" Oct 30 00:27:17.201549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.201429 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" event={"ID":"a91ac229-73ad-4732-b919-9d35d165227c","Type":"ContainerDied","Data":"436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382"} Oct 30 00:27:17.201549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.201481 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47" event={"ID":"a91ac229-73ad-4732-b919-9d35d165227c","Type":"ContainerDied","Data":"26415e3e27b653aae7579644a0b83a066b6db33e00d8c2595128c26a2c3711b0"} Oct 30 00:27:17.201549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.201503 4433 scope.go:122] "RemoveContainer" containerID="436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382" Oct 30 00:27:17.215456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.215425 4433 scope.go:122] "RemoveContainer" containerID="436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382" Oct 30 00:27:17.216208 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:27:17.216178 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382\": container with ID starting with 436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382 not found: ID does not exist" containerID="436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382" Oct 30 00:27:17.216311 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.216217 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382"} err="failed to get container status \"436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382\": rpc error: code = NotFound desc = could not find container \"436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382\": container with ID starting with 436dc086a33f3cb20159a4a2a4607f8025b918012fd4ac117988f9b501115382 not found: ID does not exist" Oct 30 00:27:17.265953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.265921 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47"] Oct 30 00:27:17.270568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.270542 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-w6d47"] Oct 30 00:27:17.791716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.791680 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:17.893829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.893789 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d"] Oct 30 00:27:17.894611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.894590 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" Oct 30 00:27:17.894727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.894612 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" Oct 30 00:27:17.894854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.894840 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91ac229-73ad-4732-b919-9d35d165227c" containerName="istio-proxy" Oct 30 00:27:17.899971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.899947 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:17.905185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.905155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:17.916155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.916125 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4500d5\"" Oct 30 00:27:17.918974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.918951 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d"] Oct 30 00:27:17.975943 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.975892 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x"] Oct 30 00:27:17.983547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.983514 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:17.985391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.985197 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bebbfd-3ad5-423f-8653-0d257c05dbef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:17.985521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.985393 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwrb\" (UniqueName: \"kubernetes.io/projected/63bebbfd-3ad5-423f-8653-0d257c05dbef-kube-api-access-hzwrb\") pod \"payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:17.989543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.989514 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x"] Oct 30 00:27:17.999201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:17.999154 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4500d5-dockercfg-g2k4l\"" Oct 30 00:27:18.147612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.147564 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmzfw\" (UniqueName: \"kubernetes.io/projected/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-kube-api-access-vmzfw\") pod \"payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:18.148115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.147637 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bebbfd-3ad5-423f-8653-0d257c05dbef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:18.148115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.147703 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwrb\" (UniqueName: \"kubernetes.io/projected/63bebbfd-3ad5-423f-8653-0d257c05dbef-kube-api-access-hzwrb\") pod \"payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:18.148115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.147774 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:18.166114 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.166077 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bebbfd-3ad5-423f-8653-0d257c05dbef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:18.194365 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.194332 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwrb\" (UniqueName: \"kubernetes.io/projected/63bebbfd-3ad5-423f-8653-0d257c05dbef-kube-api-access-hzwrb\") pod \"payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:18.217662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.217629 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:18.268609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.268071 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmzfw\" (UniqueName: \"kubernetes.io/projected/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-kube-api-access-vmzfw\") pod \"payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:18.268609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.268292 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:18.269489 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.269145 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:18.296963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.296315 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmzfw\" (UniqueName: \"kubernetes.io/projected/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-kube-api-access-vmzfw\") pod \"payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:18.309739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.309551 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:18.542173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.542138 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d"] Oct 30 00:27:18.690245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:18.690213 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x"] Oct 30 00:27:18.690676 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:27:18.690645 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4e6491_6a6c_4e8f_a9d7_ac2b59163c2f.slice/crio-d8b64b783407c0bb90cfb85faa5fa9bae745887bde7451748407057275b0565c WatchSource:0}: Error finding container d8b64b783407c0bb90cfb85faa5fa9bae745887bde7451748407057275b0565c: Status 404 returned error can't find the container with id d8b64b783407c0bb90cfb85faa5fa9bae745887bde7451748407057275b0565c Oct 30 00:27:19.181249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.181212 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91ac229-73ad-4732-b919-9d35d165227c" path="/var/lib/kubelet/pods/a91ac229-73ad-4732-b919-9d35d165227c/volumes" Oct 30 00:27:19.223768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.223734 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" event={"ID":"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f","Type":"ContainerStarted","Data":"5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf"} Oct 30 00:27:19.223984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.223789 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" event={"ID":"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f","Type":"ContainerStarted","Data":"d8b64b783407c0bb90cfb85faa5fa9bae745887bde7451748407057275b0565c"} Oct 30 00:27:19.223984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.223821 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:19.225541 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.225510 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" event={"ID":"63bebbfd-3ad5-423f-8653-0d257c05dbef","Type":"ContainerStarted","Data":"21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31"} Oct 30 00:27:19.225679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.225567 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" event={"ID":"63bebbfd-3ad5-423f-8653-0d257c05dbef","Type":"ContainerStarted","Data":"bb7675a31539fcd00fcdbf3213642505a9e4d62ded10508509b2138b9fb61e7f"} Oct 30 00:27:19.225679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.225614 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:19.265343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.265288 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" podStartSLOduration=2.265273234 podStartE2EDuration="2.265273234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:19.264809771 +0000 UTC m=+1348.198878306" watchObservedRunningTime="2026-10-30 00:27:19.265273234 +0000 UTC m=+1348.199341818" Oct 30 00:27:19.306928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.306869 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" podStartSLOduration=2.306853379 podStartE2EDuration="2.306853379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:19.305858902 +0000 UTC m=+1348.239927438" watchObservedRunningTime="2026-10-30 00:27:19.306853379 +0000 UTC m=+1348.240921914" Oct 30 00:27:19.316524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.316488 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:19.701601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:19.701542 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:20.677410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:20.677377 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:21.544035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:21.543980 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:22.574082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:22.573985 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:22.720119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:22.720087 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:23.047693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:23.047607 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:23.192133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:23.168597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:23.303112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:23.303011 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:23.420692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:23.420655 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:25.093485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:25.093442 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:25.191015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:25.190986 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:27:26.629384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:26.629346 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:28.070255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:28.070222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:28.456835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:28.456741 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:28.456835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:28.456822 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:29.448727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:29.448690 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:30.237732 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:30.237699 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:27:30.237925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:30.237776 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:27:30.765628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:30.765512 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:30.884794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:30.884756 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:31.251631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.251599 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:31.360315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.360283 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:31.453305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.453221 4433 generic.go:356] "Generic (PLEG): container finished" podID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerID="a78292dcc1fa741ea73e20651f88982aee80a5b446108df76aac0c5ea77ab098" exitCode=0 Oct 30 00:27:31.453305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.453295 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" event={"ID":"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3","Type":"ContainerDied","Data":"a78292dcc1fa741ea73e20651f88982aee80a5b446108df76aac0c5ea77ab098"} Oct 30 00:27:31.454054 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.454022 4433 scope.go:122] "RemoveContainer" containerID="a78292dcc1fa741ea73e20651f88982aee80a5b446108df76aac0c5ea77ab098" Oct 30 00:27:31.456109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.456076 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:27:31.456243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.456155 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:27:31.456243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.456188 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:27:31.538631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:31.538420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:32.480283 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:32.480241 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" event={"ID":"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3","Type":"ContainerStarted","Data":"3de5cd8b57485279691d47a36940db6ad1e2cb12400cbf99b503766a735adf10"} Oct 30 00:27:32.782663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:32.781696 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:27:32.782663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:32.781843 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:27:32.782663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:32.781888 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 00:27:33.089141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.088228 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:33.202313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.202276 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:33.216847 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.216819 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb"] Oct 30 00:27:33.220289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.220271 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:33.237519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.237496 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d3e13d\"" Oct 30 00:27:33.243532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.243509 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb"] Oct 30 00:27:33.278597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.278206 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwwv\" (UniqueName: \"kubernetes.io/projected/2f747d08-31d5-4400-be08-f8558e5f15ea-kube-api-access-sqwwv\") pod \"payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:33.278597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.278501 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f747d08-31d5-4400-be08-f8558e5f15ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:33.328614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.327938 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d3e13d-dockercfg-6s9vl\"" Oct 30 00:27:33.409648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.407453 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f747d08-31d5-4400-be08-f8558e5f15ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:33.439600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.418014 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwwv\" (UniqueName: \"kubernetes.io/projected/2f747d08-31d5-4400-be08-f8558e5f15ea-kube-api-access-sqwwv\") pod \"payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:33.440248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.440185 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f747d08-31d5-4400-be08-f8558e5f15ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:33.465609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.456118 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:33.551014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.550978 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwwv\" (UniqueName: \"kubernetes.io/projected/2f747d08-31d5-4400-be08-f8558e5f15ea-kube-api-access-sqwwv\") pod \"payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:33.610599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.609170 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:33.789192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.789090 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:27:33.789192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.789157 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 00:27:33.833830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:33.833794 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:34.184156 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:27:34.183880 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f747d08_31d5_4400_be08_f8558e5f15ea.slice/crio-c2dd0232ff631735c6aa1c79fcc77ae961f1c94832d357d3b897d5a374416a85 WatchSource:0}: Error finding container c2dd0232ff631735c6aa1c79fcc77ae961f1c94832d357d3b897d5a374416a85: Status 404 returned error can't find the container with id c2dd0232ff631735c6aa1c79fcc77ae961f1c94832d357d3b897d5a374416a85 Oct 30 00:27:34.184156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.184049 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb"] Oct 30 00:27:34.541282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.541203 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" event={"ID":"2f747d08-31d5-4400-be08-f8558e5f15ea","Type":"ContainerStarted","Data":"e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df"} Oct 30 00:27:34.541282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.541258 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" event={"ID":"2f747d08-31d5-4400-be08-f8558e5f15ea","Type":"ContainerStarted","Data":"c2dd0232ff631735c6aa1c79fcc77ae961f1c94832d357d3b897d5a374416a85"} Oct 30 00:27:34.541505 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.541407 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:34.601208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.601147 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" podStartSLOduration=1.6011267710000001 podStartE2EDuration="1.601126771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:34.599956175 +0000 UTC m=+1363.534024711" watchObservedRunningTime="2026-10-30 00:27:34.601126771 +0000 UTC m=+1363.535195308" Oct 30 00:27:34.784072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.784020 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:27:34.784256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.784118 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 00:27:34.999014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:34.998973 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:35.781782 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:35.781728 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:27:35.782479 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:35.782423 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 00:27:36.003160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:36.003119 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:36.231323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:36.231268 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:36.339739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:36.339695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:36.781601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:36.780541 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:27:36.781601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:36.780656 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 00:27:36.795614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:36.792030 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:37.137672 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:27:37.137623 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:27:37.776407 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:37.776363 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:27:37.776617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:37.776452 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 00:27:38.371000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:38.370944 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:38.486394 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:38.486359 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:38.780612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:38.780521 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:27:38.780937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:38.780916 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:27:38.781808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:38.781785 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:27:39.751158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.751075 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k"] Oct 30 00:27:39.755868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.755845 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.790402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.783728 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-metrics-certs\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.790402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.783996 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.790402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.784218 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9n4b\" (UniqueName: \"kubernetes.io/projected/6def2200-c6a3-41b1-b528-959c54018a4d-kube-api-access-z9n4b\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.790402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.784247 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k"] Oct 30 00:27:39.790402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.784311 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6def2200-c6a3-41b1-b528-959c54018a4d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.790402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.786142 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-00df5f-serving-cert\"" Oct 30 00:27:39.829418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.829383 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:39.885481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.885443 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6def2200-c6a3-41b1-b528-959c54018a4d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.885728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.885698 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-metrics-certs\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.885932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.885908 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.886111 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.886084 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n4b\" (UniqueName: \"kubernetes.io/projected/6def2200-c6a3-41b1-b528-959c54018a4d-kube-api-access-z9n4b\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.886481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.886298 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6def2200-c6a3-41b1-b528-959c54018a4d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.888795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.888767 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.888954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.888933 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-metrics-certs\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.910773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.910745 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n4b\" (UniqueName: \"kubernetes.io/projected/6def2200-c6a3-41b1-b528-959c54018a4d-kube-api-access-z9n4b\") pod \"maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:39.944259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:39.944212 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:40.049634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:40.049549 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:40.089171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:40.089139 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:40.420979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:40.420942 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k"] Oct 30 00:27:40.426743 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:27:40.426713 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6def2200_c6a3_41b1_b528_959c54018a4d.slice/crio-4b71b6b6f86d6af334515ee0125075f5cf3629e56293ac23e6496e8a9def4798 WatchSource:0}: Error finding container 4b71b6b6f86d6af334515ee0125075f5cf3629e56293ac23e6496e8a9def4798: Status 404 returned error can't find the container with id 4b71b6b6f86d6af334515ee0125075f5cf3629e56293ac23e6496e8a9def4798 Oct 30 00:27:40.428822 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:40.428797 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:27:40.622758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:40.622716 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" event={"ID":"6def2200-c6a3-41b1-b528-959c54018a4d","Type":"ContainerStarted","Data":"4b71b6b6f86d6af334515ee0125075f5cf3629e56293ac23e6496e8a9def4798"} Oct 30 00:27:41.384311 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:41.384274 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:41.634001 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:41.633944 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" event={"ID":"6def2200-c6a3-41b1-b528-959c54018a4d","Type":"ContainerStarted","Data":"ee362b5d1990be71fc93fcfbdcd23af5c2ac002e8f6df02271f180df941c15f7"} Oct 30 00:27:41.634217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:41.634127 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:41.678162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:41.678071 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" podStartSLOduration=2.3550621720000002 podStartE2EDuration="2.678057039s" totalImagesPullingTime="322.994867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:41.675982277 +0000 UTC m=+1370.610050827" watchObservedRunningTime="2026-10-30 00:27:41.678057039 +0000 UTC m=+1370.612125574" Oct 30 00:27:42.798657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.798619 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km"] Oct 30 00:27:42.803187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.803163 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.827320 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.827291 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km"] Oct 30 00:27:42.828869 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.828840 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-20d942-serving-cert\"" Oct 30 00:27:42.849682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.849648 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-metrics-certs\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.849848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.849721 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6qtn\" (UniqueName: \"kubernetes.io/projected/77773510-35f2-4b2c-a884-a381ca014eeb-kube-api-access-v6qtn\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.849994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.849957 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77773510-35f2-4b2c-a884-a381ca014eeb-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.850069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.850046 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.879356 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.879325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:42.953053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.953006 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-metrics-certs\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.954429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.954396 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6qtn\" (UniqueName: \"kubernetes.io/projected/77773510-35f2-4b2c-a884-a381ca014eeb-kube-api-access-v6qtn\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.954638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.954619 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77773510-35f2-4b2c-a884-a381ca014eeb-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.954836 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.954813 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.955158 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:27:42.955134 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-serving-cert: secret "maas-api-e2e-praxis-b-20d942-serving-cert" not found Oct 30 00:27:42.955278 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:27:42.955225 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls podName:77773510-35f2-4b2c-a884-a381ca014eeb nodeName:}" failed. No retries permitted until 2026-10-30 00:27:43.455203015 +0000 UTC m=+1372.389271529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls") pod "maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" (UID: "77773510-35f2-4b2c-a884-a381ca014eeb") : secret "maas-api-e2e-praxis-b-20d942-serving-cert" not found Oct 30 00:27:42.963307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.963282 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77773510-35f2-4b2c-a884-a381ca014eeb-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.965961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.964584 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-metrics-certs\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:42.984513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.984436 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:42.987361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:42.987333 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6qtn\" (UniqueName: \"kubernetes.io/projected/77773510-35f2-4b2c-a884-a381ca014eeb-kube-api-access-v6qtn\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:43.098925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:43.098877 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:43.463274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:43.463190 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:43.465652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:43.465630 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:43.719069 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:43.718990 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:44.030942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:44.030846 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km"] Oct 30 00:27:44.036703 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:27:44.036667 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77773510_35f2_4b2c_a884_a381ca014eeb.slice/crio-c26395052a25cc463775aa592d9b65a5a039451253f65aa82f26566b7cb2237c WatchSource:0}: Error finding container c26395052a25cc463775aa592d9b65a5a039451253f65aa82f26566b7cb2237c: Status 404 returned error can't find the container with id c26395052a25cc463775aa592d9b65a5a039451253f65aa82f26566b7cb2237c Oct 30 00:27:44.500536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:44.500483 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:44.660969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:44.660881 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" event={"ID":"77773510-35f2-4b2c-a884-a381ca014eeb","Type":"ContainerStarted","Data":"a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2"} Oct 30 00:27:44.660969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:44.660951 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" event={"ID":"77773510-35f2-4b2c-a884-a381ca014eeb","Type":"ContainerStarted","Data":"c26395052a25cc463775aa592d9b65a5a039451253f65aa82f26566b7cb2237c"} Oct 30 00:27:44.661142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:44.660981 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:44.708798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:44.708749 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" podStartSLOduration=2.350278723 podStartE2EDuration="2.708735267s" totalImagesPullingTime="358.456544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:44.704620684 +0000 UTC m=+1373.638689219" watchObservedRunningTime="2026-10-30 00:27:44.708735267 +0000 UTC m=+1373.642803803" Oct 30 00:27:45.553216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:45.553182 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:27:46.371841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:46.371809 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:47.648140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:47.648107 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:27:47.737084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:47.737045 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:49.090684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:49.090632 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:49.204633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:49.204556 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:50.587297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:50.587256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:50.676046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:50.676011 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:27:52.108973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:52.108942 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:54.097865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:54.097818 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:55.022842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:55.022811 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:55.240640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:55.240598 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:56.114437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:56.114401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:57.570403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:57.570371 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:27:58.998428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:27:58.998397 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:00.355197 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:00.355156 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:01.782710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:01.782677 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:03.375337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:03.375298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:04.481817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:04.481769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:04.784978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:04.784860 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:05.461203 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.461168 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt"] Oct 30 00:28:05.466977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.466928 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.488810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.488779 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d3e13d-serving-cert\"" Oct 30 00:28:05.494556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.494526 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt"] Oct 30 00:28:05.541732 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.541695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:05.572196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.572137 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9sg\" (UniqueName: \"kubernetes.io/projected/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-kube-api-access-lx9sg\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.572369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.572334 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.572463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.572383 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.572463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.572430 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.649833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.649794 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:05.674222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.674188 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9sg\" (UniqueName: \"kubernetes.io/projected/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-kube-api-access-lx9sg\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.674373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.674313 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.674373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.674355 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.674509 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.674398 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.675081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.675055 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.676880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.676858 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.677242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.677225 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.698186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.698158 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9sg\" (UniqueName: \"kubernetes.io/projected/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-kube-api-access-lx9sg\") pod \"maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:05.783702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:05.783614 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:06.104266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:06.104238 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt"] Oct 30 00:28:06.104417 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:28:06.104391 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ac6944_cb98_4e2f_a35e_4662ecf59e41.slice/crio-9fa1fe5920d9e74493f4fc37cc9059f3968addada1c5d5395fcce5396b6842a0 WatchSource:0}: Error finding container 9fa1fe5920d9e74493f4fc37cc9059f3968addada1c5d5395fcce5396b6842a0: Status 404 returned error can't find the container with id 9fa1fe5920d9e74493f4fc37cc9059f3968addada1c5d5395fcce5396b6842a0 Oct 30 00:28:06.858365 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:06.858321 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" event={"ID":"c5ac6944-cb98-4e2f-a35e-4662ecf59e41","Type":"ContainerStarted","Data":"d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130"} Oct 30 00:28:06.858838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:06.858394 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:06.858838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:06.858419 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" event={"ID":"c5ac6944-cb98-4e2f-a35e-4662ecf59e41","Type":"ContainerStarted","Data":"9fa1fe5920d9e74493f4fc37cc9059f3968addada1c5d5395fcce5396b6842a0"} Oct 30 00:28:06.913517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:06.913457 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" podStartSLOduration=1.573561798 podStartE2EDuration="1.913439445s" totalImagesPullingTime="339.877647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:06.903917416 +0000 UTC m=+1395.837985952" watchObservedRunningTime="2026-10-30 00:28:06.913439445 +0000 UTC m=+1395.847507979" Oct 30 00:28:07.066918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:07.066877 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:07.104876 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:07.104827 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:28:07.276382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:07.276304 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2"] Oct 30 00:28:07.276630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:07.276588 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" containerID="cri-o://2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f" gracePeriod=30 Oct 30 00:28:07.286529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:07.286500 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn"] Oct 30 00:28:07.286740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:07.286716 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" containerID="cri-o://5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de" gracePeriod=30 Oct 30 00:28:08.675682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:08.674476 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:10.578776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:10.578741 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:11.862159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:11.862122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:12.585759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.585731 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:28:12.681641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.681615 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:28:12.709558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.709523 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-data\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.709804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.709785 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-certs\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.709872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.709831 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-socket\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.709872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.709864 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-podinfo\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.709976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.709823 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-data" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.709976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.709895 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-credential-socket\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.710115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.709982 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-token\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-token\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.710115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.710021 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-certs" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.710115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.710075 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-kube-api-access-gm6pc\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-kube-api-access-gm6pc\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.710218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.710130 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-envoy\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.710263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.710234 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istiod-ca-cert\") pod \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\" (UID: \"9d148256-ff4d-48eb-b0a6-1c28033d6b51\") " Oct 30 00:28:12.710263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.710241 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-credential-socket" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.710819 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.710787 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istiod-ca-cert" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:12.711027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.711007 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-socket" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.711368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.711345 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.711466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.711376 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.711466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.711391 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.711466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.711405 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.711466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.711418 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.712377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.712357 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-envoy" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.712650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.712625 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-podinfo" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:28:12.712736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.712670 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-token" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:12.712736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.712690 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-kube-api-access-gm6pc" pod "9d148256-ff4d-48eb-b0a6-1c28033d6b51" (UID: "9d148256-ff4d-48eb-b0a6-1c28033d6b51"). InnerVolumeSpecName "kube-api-access-gm6pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:12.812440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812415 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-envoy\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812468 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-kube-api-access-4jns6\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-kube-api-access-4jns6\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812503 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306325e3-eb7b-4199-93f1-27016cd6e3ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306325e3-eb7b-4199-93f1-27016cd6e3ff-istiod-ca-cert\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812534 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-certs\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812675 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-socket\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812741 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-data\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812869 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-token\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-token\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812904 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-podinfo\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.812955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812931 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-credential-socket\") pod \"306325e3-eb7b-4199-93f1-27016cd6e3ff\" (UID: \"306325e3-eb7b-4199-93f1-27016cd6e3ff\") " Oct 30 00:28:12.813053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.812940 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306325e3-eb7b-4199-93f1-27016cd6e3ff-istiod-ca-cert" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:12.813102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813078 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-data" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.813153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813139 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-socket" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.813563 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813362 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-credential-socket" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.813563 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813543 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-certs" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.813948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813922 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6pc\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-kube-api-access-gm6pc\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.813948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813953 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813968 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813981 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306325e3-eb7b-4199-93f1-27016cd6e3ff-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.813995 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.814008 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.814021 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.814034 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.814047 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d148256-ff4d-48eb-b0a6-1c28033d6b51-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.814834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.814807 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-envoy" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:12.814983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.814965 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-kube-api-access-4jns6" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "kube-api-access-4jns6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:12.815249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.815231 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-podinfo" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:28:12.815505 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.815484 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-token" pod "306325e3-eb7b-4199-93f1-27016cd6e3ff" (UID: "306325e3-eb7b-4199-93f1-27016cd6e3ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:12.873128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.873098 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:28:12.911414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.911372 4433 generic.go:356] "Generic (PLEG): container finished" podID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerID="2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f" exitCode=0 Oct 30 00:28:12.911619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.911415 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" event={"ID":"9d148256-ff4d-48eb-b0a6-1c28033d6b51","Type":"ContainerDied","Data":"2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f"} Oct 30 00:28:12.911619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.911436 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" Oct 30 00:28:12.911619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.911480 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2" event={"ID":"9d148256-ff4d-48eb-b0a6-1c28033d6b51","Type":"ContainerDied","Data":"4c8fe205c8627c304af2c61dcec1391186aba79322c9b4ddd8aa10f2f91cf184"} Oct 30 00:28:12.911619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.911503 4433 scope.go:122] "RemoveContainer" containerID="2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f" Oct 30 00:28:12.913129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.913100 4433 generic.go:356] "Generic (PLEG): container finished" podID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerID="5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de" exitCode=0 Oct 30 00:28:12.913235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.913156 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" Oct 30 00:28:12.913235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.913175 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" event={"ID":"306325e3-eb7b-4199-93f1-27016cd6e3ff","Type":"ContainerDied","Data":"5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de"} Oct 30 00:28:12.913235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.913223 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn" event={"ID":"306325e3-eb7b-4199-93f1-27016cd6e3ff","Type":"ContainerDied","Data":"3aec28174c253eacdf9ebcce0c3de254a7725ab4220bd0dd29a4221d62517f99"} Oct 30 00:28:12.916367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.916344 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jns6\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-kube-api-access-4jns6\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.916367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.916367 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.916550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.916377 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.916550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.916385 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306325e3-eb7b-4199-93f1-27016cd6e3ff-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:12.925846 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.925821 4433 scope.go:122] "RemoveContainer" containerID="2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f" Oct 30 00:28:12.926184 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:12.926151 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f\": container with ID starting with 2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f not found: ID does not exist" containerID="2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f" Oct 30 00:28:12.926302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.926193 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f"} err="failed to get container status \"2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f\": rpc error: code = NotFound desc = could not find container \"2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f\": container with ID starting with 2676e3fb4d581a04dcfd24419d35269849e389d711d76366980b2b993df2256f not found: ID does not exist" Oct 30 00:28:12.926302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.926216 4433 scope.go:122] "RemoveContainer" containerID="5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de" Oct 30 00:28:12.938891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.938868 4433 scope.go:122] "RemoveContainer" containerID="5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de" Oct 30 00:28:12.939214 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:12.939178 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de\": container with ID starting with 5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de not found: ID does not exist" containerID="5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de" Oct 30 00:28:12.939339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.939215 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de"} err="failed to get container status \"5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de\": rpc error: code = NotFound desc = could not find container \"5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de\": container with ID starting with 5ff56dde8ce155b8f195907ce04e79a4bcaba25921f116cef9d5a7ffa34291de not found: ID does not exist" Oct 30 00:28:12.992231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.992201 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2"] Oct 30 00:28:12.996258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:12.996236 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ts2j2"] Oct 30 00:28:13.041419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:13.041392 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn"] Oct 30 00:28:13.046142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:13.045930 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-48vhn"] Oct 30 00:28:13.185158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:13.184137 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" path="/var/lib/kubelet/pods/306325e3-eb7b-4199-93f1-27016cd6e3ff/volumes" Oct 30 00:28:13.185158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:13.185085 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" path="/var/lib/kubelet/pods/9d148256-ff4d-48eb-b0a6-1c28033d6b51/volumes" Oct 30 00:28:13.310190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:13.310154 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:13.926933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:13.926892 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:14.749011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:14.748983 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:16.398317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:16.398284 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:17.974727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:17.974685 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:18.139729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:18.139700 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log" Oct 30 00:28:18.339619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:18.339597 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log" Oct 30 00:28:19.582496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:19.582459 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:20.356431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:20.356380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:20.458351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:20.458304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:21.677741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:21.677706 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:22.334467 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:22.334432 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:23.227046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:23.227007 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:24.793884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:24.793849 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:25.657723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:25.657691 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw"] Oct 30 00:28:25.737975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:25.737930 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:25.844763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:25.844681 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:26.420233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:26.420158 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:27.629727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:27.629688 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:28.058559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:28.058530 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:30.015349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:30.015311 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:30.658166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:30.658126 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" podUID="867d9d08-9c17-422a-8643-3127b4aa8589" containerName="payload-pre-processing" containerID="cri-o://19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2" gracePeriod=25 Oct 30 00:28:30.957697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:30.957672 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:28:31.025807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.025768 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867d9d08-9c17-422a-8643-3127b4aa8589-kube-api-access-hmt59\" (UniqueName: \"kubernetes.io/projected/867d9d08-9c17-422a-8643-3127b4aa8589-kube-api-access-hmt59\") pod \"867d9d08-9c17-422a-8643-3127b4aa8589\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " Oct 30 00:28:31.026502 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.026003 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867d9d08-9c17-422a-8643-3127b4aa8589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867d9d08-9c17-422a-8643-3127b4aa8589-plugins-config-volume\") pod \"867d9d08-9c17-422a-8643-3127b4aa8589\" (UID: \"867d9d08-9c17-422a-8643-3127b4aa8589\") " Oct 30 00:28:31.026502 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.026408 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867d9d08-9c17-422a-8643-3127b4aa8589-plugins-config-volume" pod "867d9d08-9c17-422a-8643-3127b4aa8589" (UID: "867d9d08-9c17-422a-8643-3127b4aa8589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:31.026910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.026881 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867d9d08-9c17-422a-8643-3127b4aa8589-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:31.028374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.028351 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867d9d08-9c17-422a-8643-3127b4aa8589-kube-api-access-hmt59" pod "867d9d08-9c17-422a-8643-3127b4aa8589" (UID: "867d9d08-9c17-422a-8643-3127b4aa8589"). InnerVolumeSpecName "kube-api-access-hmt59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:31.059152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.059112 4433 generic.go:356] "Generic (PLEG): container finished" podID="867d9d08-9c17-422a-8643-3127b4aa8589" containerID="19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2" exitCode=0 Oct 30 00:28:31.059268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.059171 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" event={"ID":"867d9d08-9c17-422a-8643-3127b4aa8589","Type":"ContainerDied","Data":"19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2"} Oct 30 00:28:31.059268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.059191 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" Oct 30 00:28:31.059268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.059224 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw" event={"ID":"867d9d08-9c17-422a-8643-3127b4aa8589","Type":"ContainerDied","Data":"2a6d97a0150bd6893c7649e13ecb79be217db1cf79758334a952bceca43d313f"} Oct 30 00:28:31.059268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.059246 4433 scope.go:122] "RemoveContainer" containerID="19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2" Oct 30 00:28:31.071005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.070985 4433 scope.go:122] "RemoveContainer" containerID="19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2" Oct 30 00:28:31.071276 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:31.071255 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2\": container with ID starting with 19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2 not found: ID does not exist" containerID="19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2" Oct 30 00:28:31.071321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.071284 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2"} err="failed to get container status \"19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2\": rpc error: code = NotFound desc = could not find container \"19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2\": container with ID starting with 19897278053a1c83cc9b9d12da0272aab0060e561a0862db215f7a3f267514c2 not found: ID does not exist" Oct 30 00:28:31.119878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.119845 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw"] Oct 30 00:28:31.130383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.130346 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cfcf7-6c865cf987-rzpzw"] Oct 30 00:28:31.133144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.133111 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmt59\" (UniqueName: \"kubernetes.io/projected/867d9d08-9c17-422a-8643-3127b4aa8589-kube-api-access-hmt59\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:31.180854 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.180777 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867d9d08-9c17-422a-8643-3127b4aa8589" path="/var/lib/kubelet/pods/867d9d08-9c17-422a-8643-3127b4aa8589/volumes" Oct 30 00:28:31.872150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:31.872114 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:32.717528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:32.717486 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x"] Oct 30 00:28:32.750999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:32.750953 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt"] Oct 30 00:28:32.857546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:32.857513 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:32.964979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:32.964886 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:33.060172 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.060130 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb"] Oct 30 00:28:33.060361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.060346 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" podUID="e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" containerName="maas-api" containerID="cri-o://4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401" gracePeriod=30 Oct 30 00:28:33.119331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.119295 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:33.336890 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.336865 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:28:33.484014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.483971 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-metrics-certs\") pod \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " Oct 30 00:28:33.495599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.487903 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-kube-api-access-qmlnk\" (UniqueName: \"kubernetes.io/projected/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-kube-api-access-qmlnk\") pod \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " Oct 30 00:28:33.495599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.487971 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-openshift-service-ca\") pod \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " Oct 30 00:28:33.495599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.488198 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls\") pod \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\" (UID: \"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7\") " Oct 30 00:28:33.496313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.487383 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-metrics-certs" pod "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" (UID: "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:33.497101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.497051 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-openshift-service-ca" pod "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" (UID: "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:33.497219 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.497132 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls" pod "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" (UID: "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:33.535763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.530733 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-kube-api-access-qmlnk" pod "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" (UID: "e0cd303d-ba4a-4dd1-a4fa-442846dff3d7"). InnerVolumeSpecName "kube-api-access-qmlnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:33.592677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.591298 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:33.592677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.591331 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlnk\" (UniqueName: \"kubernetes.io/projected/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-kube-api-access-qmlnk\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:33.592677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.591347 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:33.592677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.591362 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:33.753716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:33.753688 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:34.114651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.114612 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:34.120797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.120772 4433 generic.go:356] "Generic (PLEG): container finished" podID="e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" containerID="4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401" exitCode=0 Oct 30 00:28:34.120924 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.120824 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" Oct 30 00:28:34.120924 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.120856 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" event={"ID":"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7","Type":"ContainerDied","Data":"4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401"} Oct 30 00:28:34.120992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.120928 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb" event={"ID":"e0cd303d-ba4a-4dd1-a4fa-442846dff3d7","Type":"ContainerDied","Data":"85caefca5805365916d373e4cf649702c1ba69ca971fe17c5d982e9a7496694c"} Oct 30 00:28:34.120992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.120950 4433 scope.go:122] "RemoveContainer" containerID="4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401" Oct 30 00:28:34.132012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.131997 4433 scope.go:122] "RemoveContainer" containerID="4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401" Oct 30 00:28:34.132277 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:34.132258 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401\": container with ID starting with 4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401 not found: ID does not exist" containerID="4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401" Oct 30 00:28:34.132325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.132285 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401"} err="failed to get container status \"4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401\": rpc error: code = NotFound desc = could not find container \"4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401\": container with ID starting with 4ee864d0a751a69cf4bed5a76f9d7adaa2fd8e3bfe3e82b8c8554056b0855401 not found: ID does not exist" Oct 30 00:28:34.179311 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.179281 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb"] Oct 30 00:28:34.184485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.184462 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8b3bf0-f576d7c58-h9vwb"] Oct 30 00:28:34.216822 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.216786 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:34.762180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:34.762145 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:35.181713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:35.181673 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" path="/var/lib/kubelet/pods/e0cd303d-ba4a-4dd1-a4fa-442846dff3d7/volumes" Oct 30 00:28:35.490324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:35.490225 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:35.747504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:35.747423 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:35.850119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:35.850083 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:37.101795 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:37.101759 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:28:37.119806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:37.119780 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:37.718544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:37.718506 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" podUID="580149e6-952f-43dc-84cd-4e004513bbbc" containerName="payload-processing" containerID="cri-o://70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7" gracePeriod=25 Oct 30 00:28:37.751866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:37.751816 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" podUID="424d4f13-177e-4ef4-986d-60f788399800" containerName="payload-pre-processing" containerID="cri-o://fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d" gracePeriod=25 Oct 30 00:28:38.058984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.058938 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:28:38.087676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.087653 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:28:38.155525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.155482 4433 generic.go:356] "Generic (PLEG): container finished" podID="580149e6-952f-43dc-84cd-4e004513bbbc" containerID="70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7" exitCode=0 Oct 30 00:28:38.156060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.155546 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" Oct 30 00:28:38.156060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.155797 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" event={"ID":"580149e6-952f-43dc-84cd-4e004513bbbc","Type":"ContainerDied","Data":"70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7"} Oct 30 00:28:38.156060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.155883 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x" event={"ID":"580149e6-952f-43dc-84cd-4e004513bbbc","Type":"ContainerDied","Data":"5c7e2d3df186710f4cd7ab3443395f3b59cfef0d23a2ac29aaad9d55b4cd77fa"} Oct 30 00:28:38.156060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.155917 4433 scope.go:122] "RemoveContainer" containerID="70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7" Oct 30 00:28:38.158424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.158393 4433 generic.go:356] "Generic (PLEG): container finished" podID="424d4f13-177e-4ef4-986d-60f788399800" containerID="fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d" exitCode=0 Oct 30 00:28:38.158549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.158465 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" event={"ID":"424d4f13-177e-4ef4-986d-60f788399800","Type":"ContainerDied","Data":"fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d"} Oct 30 00:28:38.158549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.158499 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" event={"ID":"424d4f13-177e-4ef4-986d-60f788399800","Type":"ContainerDied","Data":"1dbb70fefd4d1f8570f9f125c246b23bfdb0273c169bc84ff98d06eef7a3f6bd"} Oct 30 00:28:38.158549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.158520 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt" Oct 30 00:28:38.165015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.164982 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424d4f13-177e-4ef4-986d-60f788399800-kube-api-access-4lt72\" (UniqueName: \"kubernetes.io/projected/424d4f13-177e-4ef4-986d-60f788399800-kube-api-access-4lt72\") pod \"424d4f13-177e-4ef4-986d-60f788399800\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " Oct 30 00:28:38.165126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.165035 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424d4f13-177e-4ef4-986d-60f788399800-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424d4f13-177e-4ef4-986d-60f788399800-plugins-config-volume\") pod \"424d4f13-177e-4ef4-986d-60f788399800\" (UID: \"424d4f13-177e-4ef4-986d-60f788399800\") " Oct 30 00:28:38.165807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.165782 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424d4f13-177e-4ef4-986d-60f788399800-plugins-config-volume" pod "424d4f13-177e-4ef4-986d-60f788399800" (UID: "424d4f13-177e-4ef4-986d-60f788399800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:38.166511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.166485 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/580149e6-952f-43dc-84cd-4e004513bbbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580149e6-952f-43dc-84cd-4e004513bbbc-plugins-config-volume\") pod \"580149e6-952f-43dc-84cd-4e004513bbbc\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " Oct 30 00:28:38.166669 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.166648 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580149e6-952f-43dc-84cd-4e004513bbbc-kube-api-access-wf4q2\" (UniqueName: \"kubernetes.io/projected/580149e6-952f-43dc-84cd-4e004513bbbc-kube-api-access-wf4q2\") pod \"580149e6-952f-43dc-84cd-4e004513bbbc\" (UID: \"580149e6-952f-43dc-84cd-4e004513bbbc\") " Oct 30 00:28:38.167382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.167356 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580149e6-952f-43dc-84cd-4e004513bbbc-plugins-config-volume" pod "580149e6-952f-43dc-84cd-4e004513bbbc" (UID: "580149e6-952f-43dc-84cd-4e004513bbbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:38.167485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.167436 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424d4f13-177e-4ef4-986d-60f788399800-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:38.167485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.167468 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424d4f13-177e-4ef4-986d-60f788399800-kube-api-access-4lt72" pod "424d4f13-177e-4ef4-986d-60f788399800" (UID: "424d4f13-177e-4ef4-986d-60f788399800"). InnerVolumeSpecName "kube-api-access-4lt72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:38.168746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.168724 4433 scope.go:122] "RemoveContainer" containerID="70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7" Oct 30 00:28:38.168914 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.168889 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580149e6-952f-43dc-84cd-4e004513bbbc-kube-api-access-wf4q2" pod "580149e6-952f-43dc-84cd-4e004513bbbc" (UID: "580149e6-952f-43dc-84cd-4e004513bbbc"). InnerVolumeSpecName "kube-api-access-wf4q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:38.169042 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:38.169019 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7\": container with ID starting with 70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7 not found: ID does not exist" containerID="70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7" Oct 30 00:28:38.169157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.169048 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7"} err="failed to get container status \"70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7\": rpc error: code = NotFound desc = could not find container \"70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7\": container with ID starting with 70fb646ce030d82b650d81cbb939d2dff0d3ab73f8b6024c64e2a79d6b9ee0d7 not found: ID does not exist" Oct 30 00:28:38.169157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.169070 4433 scope.go:122] "RemoveContainer" containerID="fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d" Oct 30 00:28:38.180871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.180855 4433 scope.go:122] "RemoveContainer" containerID="fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d" Oct 30 00:28:38.181168 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:38.181148 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d\": container with ID starting with fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d not found: ID does not exist" containerID="fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d" Oct 30 00:28:38.181245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.181176 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d"} err="failed to get container status \"fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d\": rpc error: code = NotFound desc = could not find container \"fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d\": container with ID starting with fde7fbeab0f17b39de722d885257dab7d81d94f7eee25d72abb830aacb15820d not found: ID does not exist" Oct 30 00:28:38.269544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.269460 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lt72\" (UniqueName: \"kubernetes.io/projected/424d4f13-177e-4ef4-986d-60f788399800-kube-api-access-4lt72\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:38.269544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.269496 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580149e6-952f-43dc-84cd-4e004513bbbc-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:38.269544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.269520 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4q2\" (UniqueName: \"kubernetes.io/projected/580149e6-952f-43dc-84cd-4e004513bbbc-kube-api-access-wf4q2\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:38.506175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.506147 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x"] Oct 30 00:28:38.520331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.520269 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8b3bf0-b5b6d9c54-6gp2x"] Oct 30 00:28:38.544322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.544267 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt"] Oct 30 00:28:38.548621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.548598 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8b3bf0-6b67c97bbc-jhfxt"] Oct 30 00:28:38.587043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:38.587011 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:39.173204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:39.173170 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:39.182416 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:39.182389 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424d4f13-177e-4ef4-986d-60f788399800" path="/var/lib/kubelet/pods/424d4f13-177e-4ef4-986d-60f788399800/volumes" Oct 30 00:28:39.182939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:39.182924 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580149e6-952f-43dc-84cd-4e004513bbbc" path="/var/lib/kubelet/pods/580149e6-952f-43dc-84cd-4e004513bbbc/volumes" Oct 30 00:28:40.617866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:40.617828 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:41.030371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:41.030289 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:41.133242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:41.133203 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:41.377460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:41.377426 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:42.304257 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:42.304221 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:42.443078 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:42.443045 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:42.679348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:42.679315 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:42.791224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:42.791191 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:43.566680 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:43.566647 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp"] Oct 30 00:28:43.567184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:43.566880 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" podUID="d51c7cd0-80c4-48cb-a6be-67054065240c" containerName="maas-api" containerID="cri-o://3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1" gracePeriod=30 Oct 30 00:28:43.635766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:43.635732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:43.944239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:43.943862 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:28:44.006741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.003522 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d51c7cd0-80c4-48cb-a6be-67054065240c-kube-api-access-jsc6r\" (UniqueName: \"kubernetes.io/projected/d51c7cd0-80c4-48cb-a6be-67054065240c-kube-api-access-jsc6r\") pod \"d51c7cd0-80c4-48cb-a6be-67054065240c\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " Oct 30 00:28:44.006741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.003651 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-maas-api-tls\") pod \"d51c7cd0-80c4-48cb-a6be-67054065240c\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " Oct 30 00:28:44.006741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.004302 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-metrics-certs\") pod \"d51c7cd0-80c4-48cb-a6be-67054065240c\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " Oct 30 00:28:44.006741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.004600 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d51c7cd0-80c4-48cb-a6be-67054065240c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51c7cd0-80c4-48cb-a6be-67054065240c-openshift-service-ca\") pod \"d51c7cd0-80c4-48cb-a6be-67054065240c\" (UID: \"d51c7cd0-80c4-48cb-a6be-67054065240c\") " Oct 30 00:28:44.006741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.006135 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51c7cd0-80c4-48cb-a6be-67054065240c-openshift-service-ca" pod "d51c7cd0-80c4-48cb-a6be-67054065240c" (UID: "d51c7cd0-80c4-48cb-a6be-67054065240c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:44.017775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.017714 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-metrics-certs" pod "d51c7cd0-80c4-48cb-a6be-67054065240c" (UID: "d51c7cd0-80c4-48cb-a6be-67054065240c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:44.019426 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.019401 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-maas-api-tls" pod "d51c7cd0-80c4-48cb-a6be-67054065240c" (UID: "d51c7cd0-80c4-48cb-a6be-67054065240c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:44.020251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.020232 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51c7cd0-80c4-48cb-a6be-67054065240c-kube-api-access-jsc6r" pod "d51c7cd0-80c4-48cb-a6be-67054065240c" (UID: "d51c7cd0-80c4-48cb-a6be-67054065240c"). InnerVolumeSpecName "kube-api-access-jsc6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:44.107190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.107113 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:44.107190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.107144 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51c7cd0-80c4-48cb-a6be-67054065240c-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:44.107190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.107154 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsc6r\" (UniqueName: \"kubernetes.io/projected/d51c7cd0-80c4-48cb-a6be-67054065240c-kube-api-access-jsc6r\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:44.107190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.107164 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51c7cd0-80c4-48cb-a6be-67054065240c-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:44.222796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.222761 4433 generic.go:356] "Generic (PLEG): container finished" podID="d51c7cd0-80c4-48cb-a6be-67054065240c" containerID="3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1" exitCode=0 Oct 30 00:28:44.222978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.222843 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" event={"ID":"d51c7cd0-80c4-48cb-a6be-67054065240c","Type":"ContainerDied","Data":"3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1"} Oct 30 00:28:44.222978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.222899 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" event={"ID":"d51c7cd0-80c4-48cb-a6be-67054065240c","Type":"ContainerDied","Data":"30ec9bdc0d7e95d18cd7115109374307ef47ebacaa4029d82c5a6e82c367e049"} Oct 30 00:28:44.222978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.222928 4433 scope.go:122] "RemoveContainer" containerID="3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1" Oct 30 00:28:44.223081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.223032 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp" Oct 30 00:28:44.235314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.235294 4433 scope.go:122] "RemoveContainer" containerID="3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1" Oct 30 00:28:44.235660 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:44.235635 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1\": container with ID starting with 3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1 not found: ID does not exist" containerID="3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1" Oct 30 00:28:44.235739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.235672 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1"} err="failed to get container status \"3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1\": rpc error: code = NotFound desc = could not find container \"3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1\": container with ID starting with 3f7f41e1684a6cde5517458be05357eb73b1224c5699d9534b8d23c57d0ac5f1 not found: ID does not exist" Oct 30 00:28:44.251053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.251021 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:44.279610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.279560 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp"] Oct 30 00:28:44.291757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.291726 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-522ea7-586d998585-6smkp"] Oct 30 00:28:44.430119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.428452 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:44.539602 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.536455 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:44.973288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:44.973140 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:45.181730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:45.181698 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d51c7cd0-80c4-48cb-a6be-67054065240c" path="/var/lib/kubelet/pods/d51c7cd0-80c4-48cb-a6be-67054065240c/volumes" Oct 30 00:28:46.195399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:46.195359 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:47.765076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:47.765037 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:49.120158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:49.120124 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:50.557142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:50.557095 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:51.283374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:51.283337 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:51.385824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:51.385791 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:51.628204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:51.628169 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:51.678996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:51.678957 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7"] Oct 30 00:28:51.679174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:51.679153 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" podUID="cd0df4b3-3098-41d8-84ca-23c5a23b3183" containerName="maas-api" containerID="cri-o://b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0" gracePeriod=30 Oct 30 00:28:51.741527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:51.741489 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:51.993128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:51.993096 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:52.007050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.007023 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:28:52.077275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.077240 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0df4b3-3098-41d8-84ca-23c5a23b3183-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0df4b3-3098-41d8-84ca-23c5a23b3183-openshift-service-ca\") pod \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " Oct 30 00:28:52.077778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.077747 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-metrics-certs\") pod \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " Oct 30 00:28:52.078015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.077990 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0df4b3-3098-41d8-84ca-23c5a23b3183-openshift-service-ca" pod "cd0df4b3-3098-41d8-84ca-23c5a23b3183" (UID: "cd0df4b3-3098-41d8-84ca-23c5a23b3183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:52.078155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.078135 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0df4b3-3098-41d8-84ca-23c5a23b3183-kube-api-access-g2bt4\" (UniqueName: \"kubernetes.io/projected/cd0df4b3-3098-41d8-84ca-23c5a23b3183-kube-api-access-g2bt4\") pod \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " Oct 30 00:28:52.078406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.078378 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-maas-api-tls\") pod \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\" (UID: \"cd0df4b3-3098-41d8-84ca-23c5a23b3183\") " Oct 30 00:28:52.079209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.079186 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0df4b3-3098-41d8-84ca-23c5a23b3183-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:52.083241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.083205 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-maas-api-tls" pod "cd0df4b3-3098-41d8-84ca-23c5a23b3183" (UID: "cd0df4b3-3098-41d8-84ca-23c5a23b3183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:52.084038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.084011 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-metrics-certs" pod "cd0df4b3-3098-41d8-84ca-23c5a23b3183" (UID: "cd0df4b3-3098-41d8-84ca-23c5a23b3183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:52.085990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.085805 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0df4b3-3098-41d8-84ca-23c5a23b3183-kube-api-access-g2bt4" pod "cd0df4b3-3098-41d8-84ca-23c5a23b3183" (UID: "cd0df4b3-3098-41d8-84ca-23c5a23b3183"). InnerVolumeSpecName "kube-api-access-g2bt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:52.107161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.107112 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:52.181690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.181655 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:52.181690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.181693 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bt4\" (UniqueName: \"kubernetes.io/projected/cd0df4b3-3098-41d8-84ca-23c5a23b3183-kube-api-access-g2bt4\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:52.181690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.181708 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0df4b3-3098-41d8-84ca-23c5a23b3183-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:28:52.296242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.296070 4433 generic.go:356] "Generic (PLEG): container finished" podID="cd0df4b3-3098-41d8-84ca-23c5a23b3183" containerID="b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0" exitCode=0 Oct 30 00:28:52.296837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.296302 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" event={"ID":"cd0df4b3-3098-41d8-84ca-23c5a23b3183","Type":"ContainerDied","Data":"b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0"} Oct 30 00:28:52.297083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.297044 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" event={"ID":"cd0df4b3-3098-41d8-84ca-23c5a23b3183","Type":"ContainerDied","Data":"fac4a78ff3461f8c15a6f3be8ddc2300220334f3fd6fa97e1add97584193b044"} Oct 30 00:28:52.297194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.297094 4433 scope.go:122] "RemoveContainer" containerID="b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0" Oct 30 00:28:52.297704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.297683 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7" Oct 30 00:28:52.318110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.315717 4433 scope.go:122] "RemoveContainer" containerID="b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0" Oct 30 00:28:52.318110 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:28:52.316070 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0\": container with ID starting with b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0 not found: ID does not exist" containerID="b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0" Oct 30 00:28:52.318110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.316108 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0"} err="failed to get container status \"b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0\": rpc error: code = NotFound desc = could not find container \"b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0\": container with ID starting with b761a6221fce1be1002199fa76b4efeb70dc7283611f30af1e466bd03572ddd0 not found: ID does not exist" Oct 30 00:28:52.318546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.318134 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:52.360088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.360035 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7"] Oct 30 00:28:52.370425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.370394 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8397b-786cb764d-5cwh7"] Oct 30 00:28:52.421774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.421738 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:52.525850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.525797 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:52.632470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.632433 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:52.930251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:52.930090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:53.046478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:53.046435 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:53.186714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:53.186637 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0df4b3-3098-41d8-84ca-23c5a23b3183" path="/var/lib/kubelet/pods/cd0df4b3-3098-41d8-84ca-23c5a23b3183/volumes" Oct 30 00:28:54.075488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:54.075454 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:54.233842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:54.233810 4433 scope.go:122] "RemoveContainer" containerID="799bdc1e85f824e02470f47312ca6ba216caa931045ab6f3daa280bd7b7f9a91" Oct 30 00:28:55.659836 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:55.659797 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:57.252588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:57.252536 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:58.803786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:58.803746 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:58.906565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:58.906523 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:28:59.235720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:28:59.235689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:00.883433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:00.883397 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:02.747661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:02.747619 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:03.775920 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:03.775875 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5"] Oct 30 00:29:03.776438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:03.776134 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" podUID="bccd33e5-4ecd-4528-b879-a460841f4e90" containerName="maas-api" containerID="cri-o://f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d" gracePeriod=30 Oct 30 00:29:03.833907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:03.833871 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:04.137648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.133807 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:29:04.260185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.260145 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:04.288603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.285858 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-metrics-certs\") pod \"bccd33e5-4ecd-4528-b879-a460841f4e90\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " Oct 30 00:29:04.288603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.285947 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccd33e5-4ecd-4528-b879-a460841f4e90-kube-api-access-8sq42\" (UniqueName: \"kubernetes.io/projected/bccd33e5-4ecd-4528-b879-a460841f4e90-kube-api-access-8sq42\") pod \"bccd33e5-4ecd-4528-b879-a460841f4e90\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " Oct 30 00:29:04.288603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.286083 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-maas-api-tls\") pod \"bccd33e5-4ecd-4528-b879-a460841f4e90\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " Oct 30 00:29:04.288603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.286268 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccd33e5-4ecd-4528-b879-a460841f4e90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccd33e5-4ecd-4528-b879-a460841f4e90-openshift-service-ca\") pod \"bccd33e5-4ecd-4528-b879-a460841f4e90\" (UID: \"bccd33e5-4ecd-4528-b879-a460841f4e90\") " Oct 30 00:29:04.288603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.287745 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccd33e5-4ecd-4528-b879-a460841f4e90-openshift-service-ca" pod "bccd33e5-4ecd-4528-b879-a460841f4e90" (UID: "bccd33e5-4ecd-4528-b879-a460841f4e90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:04.296643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.291134 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-maas-api-tls" pod "bccd33e5-4ecd-4528-b879-a460841f4e90" (UID: "bccd33e5-4ecd-4528-b879-a460841f4e90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:04.303717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.303667 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccd33e5-4ecd-4528-b879-a460841f4e90-kube-api-access-8sq42" pod "bccd33e5-4ecd-4528-b879-a460841f4e90" (UID: "bccd33e5-4ecd-4528-b879-a460841f4e90"). InnerVolumeSpecName "kube-api-access-8sq42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:04.317613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.313761 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-metrics-certs" pod "bccd33e5-4ecd-4528-b879-a460841f4e90" (UID: "bccd33e5-4ecd-4528-b879-a460841f4e90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:04.363767 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.363720 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:04.388820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.388786 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccd33e5-4ecd-4528-b879-a460841f4e90-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:04.388820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.388813 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:04.388820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.388824 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sq42\" (UniqueName: \"kubernetes.io/projected/bccd33e5-4ecd-4528-b879-a460841f4e90-kube-api-access-8sq42\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:04.389139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.388834 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bccd33e5-4ecd-4528-b879-a460841f4e90-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:04.413820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.413782 4433 generic.go:356] "Generic (PLEG): container finished" podID="bccd33e5-4ecd-4528-b879-a460841f4e90" containerID="f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d" exitCode=0 Oct 30 00:29:04.414008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.413830 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" Oct 30 00:29:04.414008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.413867 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" event={"ID":"bccd33e5-4ecd-4528-b879-a460841f4e90","Type":"ContainerDied","Data":"f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d"} Oct 30 00:29:04.414008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.413933 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5" event={"ID":"bccd33e5-4ecd-4528-b879-a460841f4e90","Type":"ContainerDied","Data":"612074ff3f5e22606cb2aa8342a745cdf71e2a88f81ed3d7a0f217d5c0d078ab"} Oct 30 00:29:04.414008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.413951 4433 scope.go:122] "RemoveContainer" containerID="f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d" Oct 30 00:29:04.424761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.424742 4433 scope.go:122] "RemoveContainer" containerID="f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d" Oct 30 00:29:04.425109 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:29:04.425088 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d\": container with ID starting with f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d not found: ID does not exist" containerID="f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d" Oct 30 00:29:04.425181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.425116 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d"} err="failed to get container status \"f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d\": rpc error: code = NotFound desc = could not find container \"f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d\": container with ID starting with f52c630d69e2b33a3df6e35677b726d0442a48ffbb602a125f8fdbfd5248ab1d not found: ID does not exist" Oct 30 00:29:04.475162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.475068 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:04.482319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.480901 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5"] Oct 30 00:29:04.486975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.486935 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cfcf7-8664d4446f-zktl5"] Oct 30 00:29:04.931208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:04.931167 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:05.181184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:05.181150 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccd33e5-4ecd-4528-b879-a460841f4e90" path="/var/lib/kubelet/pods/bccd33e5-4ecd-4528-b879-a460841f4e90/volumes" Oct 30 00:29:05.659981 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:05.657247 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc"] Oct 30 00:29:05.659981 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:05.657567 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" podUID="1b98754c-b660-4dbe-aa39-a01a1fb1fe16" containerName="maas-api" containerID="cri-o://18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b" gracePeriod=30 Oct 30 00:29:05.698214 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:05.698179 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:05.782073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:05.782037 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" podUID="1b98754c-b660-4dbe-aa39-a01a1fb1fe16" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.120:8443/health\": dial tcp 10.134.0.120:8443: connect: connection refused" Oct 30 00:29:06.008692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.007683 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:29:06.119049 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.119012 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-maas-api-tls\") pod \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " Oct 30 00:29:06.119255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.119063 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-metrics-certs\") pod \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " Oct 30 00:29:06.119255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.119226 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-kube-api-access-jvgcr\" (UniqueName: \"kubernetes.io/projected/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-kube-api-access-jvgcr\") pod \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " Oct 30 00:29:06.119385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.119333 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-openshift-service-ca\") pod \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\" (UID: \"1b98754c-b660-4dbe-aa39-a01a1fb1fe16\") " Oct 30 00:29:06.119718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.119686 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-openshift-service-ca" pod "1b98754c-b660-4dbe-aa39-a01a1fb1fe16" (UID: "1b98754c-b660-4dbe-aa39-a01a1fb1fe16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:06.120085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.120065 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:06.121187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.121166 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-maas-api-tls" pod "1b98754c-b660-4dbe-aa39-a01a1fb1fe16" (UID: "1b98754c-b660-4dbe-aa39-a01a1fb1fe16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:06.121286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.121186 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-metrics-certs" pod "1b98754c-b660-4dbe-aa39-a01a1fb1fe16" (UID: "1b98754c-b660-4dbe-aa39-a01a1fb1fe16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:06.121286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.121258 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-kube-api-access-jvgcr" pod "1b98754c-b660-4dbe-aa39-a01a1fb1fe16" (UID: "1b98754c-b660-4dbe-aa39-a01a1fb1fe16"). InnerVolumeSpecName "kube-api-access-jvgcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:06.222445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.222253 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:06.222445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.222287 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:06.222445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.222305 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgcr\" (UniqueName: \"kubernetes.io/projected/1b98754c-b660-4dbe-aa39-a01a1fb1fe16-kube-api-access-jvgcr\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:06.242342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.242305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:06.439052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.439009 4433 generic.go:356] "Generic (PLEG): container finished" podID="1b98754c-b660-4dbe-aa39-a01a1fb1fe16" containerID="18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b" exitCode=0 Oct 30 00:29:06.439256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.439071 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" event={"ID":"1b98754c-b660-4dbe-aa39-a01a1fb1fe16","Type":"ContainerDied","Data":"18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b"} Oct 30 00:29:06.439256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.439079 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" Oct 30 00:29:06.439256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.439122 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc" event={"ID":"1b98754c-b660-4dbe-aa39-a01a1fb1fe16","Type":"ContainerDied","Data":"656e7dcb5d356e35ccc854793c3538305a3ed12191c9c2f6e3dcc9f0c0e79399"} Oct 30 00:29:06.439256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.439151 4433 scope.go:122] "RemoveContainer" containerID="18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b" Oct 30 00:29:06.443485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.443452 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:06.452314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.452294 4433 scope.go:122] "RemoveContainer" containerID="18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b" Oct 30 00:29:06.452668 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:29:06.452648 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b\": container with ID starting with 18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b not found: ID does not exist" containerID="18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b" Oct 30 00:29:06.452763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.452685 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b"} err="failed to get container status \"18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b\": rpc error: code = NotFound desc = could not find container \"18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b\": container with ID starting with 18004894fe82070d92a3b91ef588ba7deebb46cace52f3306bfb194ea93ab77b not found: ID does not exist" Oct 30 00:29:06.497013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.496912 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc"] Oct 30 00:29:06.501597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.501554 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bb16a8-755ff79485-d45nc"] Oct 30 00:29:06.571595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.570659 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:06.945832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:06.945799 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:07.100550 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:29:07.100513 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:29:07.181062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:07.181026 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b98754c-b660-4dbe-aa39-a01a1fb1fe16" path="/var/lib/kubelet/pods/1b98754c-b660-4dbe-aa39-a01a1fb1fe16/volumes" Oct 30 00:29:07.879153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:07.879121 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d"] Oct 30 00:29:07.911887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:07.911852 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x"] Oct 30 00:29:08.022108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:08.022074 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:08.132475 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:08.132385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:08.476841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:08.476753 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:10.178443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:10.178411 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:11.344186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:11.344152 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:11.446215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:11.446174 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:11.705297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:11.705213 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:11.851426 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:11.851391 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:12.879765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:12.879716 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" podUID="63bebbfd-3ad5-423f-8653-0d257c05dbef" containerName="payload-processing" containerID="cri-o://21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31" gracePeriod=25 Oct 30 00:29:12.912117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:12.912085 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" podUID="8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" containerName="payload-pre-processing" containerID="cri-o://5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf" gracePeriod=25 Oct 30 00:29:13.234083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.234060 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:29:13.259260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.259231 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:29:13.341539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.341497 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-kube-api-access-vmzfw\" (UniqueName: \"kubernetes.io/projected/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-kube-api-access-vmzfw\") pod \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " Oct 30 00:29:13.341753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.341683 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-plugins-config-volume\") pod \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\" (UID: \"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f\") " Oct 30 00:29:13.341836 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.341782 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bebbfd-3ad5-423f-8653-0d257c05dbef-kube-api-access-hzwrb\" (UniqueName: \"kubernetes.io/projected/63bebbfd-3ad5-423f-8653-0d257c05dbef-kube-api-access-hzwrb\") pod \"63bebbfd-3ad5-423f-8653-0d257c05dbef\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " Oct 30 00:29:13.341976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.341959 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bebbfd-3ad5-423f-8653-0d257c05dbef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bebbfd-3ad5-423f-8653-0d257c05dbef-plugins-config-volume\") pod \"63bebbfd-3ad5-423f-8653-0d257c05dbef\" (UID: \"63bebbfd-3ad5-423f-8653-0d257c05dbef\") " Oct 30 00:29:13.342073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.342044 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-plugins-config-volume" pod "8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" (UID: "8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:13.342341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.342320 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bebbfd-3ad5-423f-8653-0d257c05dbef-plugins-config-volume" pod "63bebbfd-3ad5-423f-8653-0d257c05dbef" (UID: "63bebbfd-3ad5-423f-8653-0d257c05dbef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:13.342717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.342695 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:13.342807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.342721 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bebbfd-3ad5-423f-8653-0d257c05dbef-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:13.344140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.344109 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bebbfd-3ad5-423f-8653-0d257c05dbef-kube-api-access-hzwrb" pod "63bebbfd-3ad5-423f-8653-0d257c05dbef" (UID: "63bebbfd-3ad5-423f-8653-0d257c05dbef"). InnerVolumeSpecName "kube-api-access-hzwrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:13.344376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.344347 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-kube-api-access-vmzfw" pod "8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" (UID: "8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f"). InnerVolumeSpecName "kube-api-access-vmzfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:13.444795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.444709 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwrb\" (UniqueName: \"kubernetes.io/projected/63bebbfd-3ad5-423f-8653-0d257c05dbef-kube-api-access-hzwrb\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:13.444795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.444740 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmzfw\" (UniqueName: \"kubernetes.io/projected/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f-kube-api-access-vmzfw\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:13.499933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.499901 4433 generic.go:356] "Generic (PLEG): container finished" podID="8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" containerID="5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf" exitCode=0 Oct 30 00:29:13.500094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.499947 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" Oct 30 00:29:13.500094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.499990 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" event={"ID":"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f","Type":"ContainerDied","Data":"5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf"} Oct 30 00:29:13.500094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.500050 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x" event={"ID":"8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f","Type":"ContainerDied","Data":"d8b64b783407c0bb90cfb85faa5fa9bae745887bde7451748407057275b0565c"} Oct 30 00:29:13.500094 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.500078 4433 scope.go:122] "RemoveContainer" containerID="5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf" Oct 30 00:29:13.501758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.501731 4433 generic.go:356] "Generic (PLEG): container finished" podID="63bebbfd-3ad5-423f-8653-0d257c05dbef" containerID="21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31" exitCode=0 Oct 30 00:29:13.501859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.501794 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" Oct 30 00:29:13.501859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.501813 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" event={"ID":"63bebbfd-3ad5-423f-8653-0d257c05dbef","Type":"ContainerDied","Data":"21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31"} Oct 30 00:29:13.501954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.501869 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d" event={"ID":"63bebbfd-3ad5-423f-8653-0d257c05dbef","Type":"ContainerDied","Data":"bb7675a31539fcd00fcdbf3213642505a9e4d62ded10508509b2138b9fb61e7f"} Oct 30 00:29:13.514177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.514156 4433 scope.go:122] "RemoveContainer" containerID="5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf" Oct 30 00:29:13.514547 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:29:13.514522 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf\": container with ID starting with 5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf not found: ID does not exist" containerID="5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf" Oct 30 00:29:13.514709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.514559 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf"} err="failed to get container status \"5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf\": rpc error: code = NotFound desc = could not find container \"5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf\": container with ID starting with 5c46e07b1049e4aec1c4e3ce6ed2a5d40a5b682293302993154304b5385ca0cf not found: ID does not exist" Oct 30 00:29:13.514709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.514596 4433 scope.go:122] "RemoveContainer" containerID="21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31" Oct 30 00:29:13.526931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.526905 4433 scope.go:122] "RemoveContainer" containerID="21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31" Oct 30 00:29:13.527222 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:29:13.527199 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31\": container with ID starting with 21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31 not found: ID does not exist" containerID="21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31" Oct 30 00:29:13.527264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.527231 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31"} err="failed to get container status \"21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31\": rpc error: code = NotFound desc = could not find container \"21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31\": container with ID starting with 21a9fc12a48ef2790891068ef90061a6feadf800b473a3665006f3a650e15b31 not found: ID does not exist" Oct 30 00:29:13.561353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.561308 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x"] Oct 30 00:29:13.569076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.569039 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:13.573740 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.573710 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4500d5-8859c7cff-q6s5x"] Oct 30 00:29:13.616201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.616150 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d"] Oct 30 00:29:13.620599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:13.620554 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4500d5-85f4486f78-xvl4d"] Oct 30 00:29:14.068645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:14.068606 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:14.171327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:14.171228 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:14.449988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:14.449901 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:15.155052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:15.155016 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:15.190997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:15.190954 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bebbfd-3ad5-423f-8653-0d257c05dbef" path="/var/lib/kubelet/pods/63bebbfd-3ad5-423f-8653-0d257c05dbef/volumes" Oct 30 00:29:15.191746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:15.191720 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" path="/var/lib/kubelet/pods/8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f/volumes" Oct 30 00:29:16.420225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:16.420191 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k"] Oct 30 00:29:16.420808 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:16.420478 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" podUID="6def2200-c6a3-41b1-b528-959c54018a4d" containerName="maas-api" containerID="cri-o://ee362b5d1990be71fc93fcfbdcd23af5c2ac002e8f6df02271f180df941c15f7" gracePeriod=30 Oct 30 00:29:16.478666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:16.478559 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:16.621232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:16.621167 4433 generic.go:356] "Generic (PLEG): container finished" podID="6def2200-c6a3-41b1-b528-959c54018a4d" containerID="ee362b5d1990be71fc93fcfbdcd23af5c2ac002e8f6df02271f180df941c15f7" exitCode=0 Oct 30 00:29:16.622254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:16.622220 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" event={"ID":"6def2200-c6a3-41b1-b528-959c54018a4d","Type":"ContainerDied","Data":"ee362b5d1990be71fc93fcfbdcd23af5c2ac002e8f6df02271f180df941c15f7"} Oct 30 00:29:16.816761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:16.816733 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:29:16.958650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:16.958616 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:17.008759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.008727 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6def2200-c6a3-41b1-b528-959c54018a4d-kube-api-access-z9n4b\" (UniqueName: \"kubernetes.io/projected/6def2200-c6a3-41b1-b528-959c54018a4d-kube-api-access-z9n4b\") pod \"6def2200-c6a3-41b1-b528-959c54018a4d\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " Oct 30 00:29:17.008906 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.008880 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-metrics-certs\") pod \"6def2200-c6a3-41b1-b528-959c54018a4d\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " Oct 30 00:29:17.008961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.008948 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-maas-api-tls\") pod \"6def2200-c6a3-41b1-b528-959c54018a4d\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " Oct 30 00:29:17.009061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.009046 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6def2200-c6a3-41b1-b528-959c54018a4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6def2200-c6a3-41b1-b528-959c54018a4d-openshift-service-ca\") pod \"6def2200-c6a3-41b1-b528-959c54018a4d\" (UID: \"6def2200-c6a3-41b1-b528-959c54018a4d\") " Oct 30 00:29:17.009502 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.009473 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6def2200-c6a3-41b1-b528-959c54018a4d-openshift-service-ca" pod "6def2200-c6a3-41b1-b528-959c54018a4d" (UID: "6def2200-c6a3-41b1-b528-959c54018a4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:17.009828 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.009805 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6def2200-c6a3-41b1-b528-959c54018a4d-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.011408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.011385 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-maas-api-tls" pod "6def2200-c6a3-41b1-b528-959c54018a4d" (UID: "6def2200-c6a3-41b1-b528-959c54018a4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:17.011521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.011442 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6def2200-c6a3-41b1-b528-959c54018a4d-kube-api-access-z9n4b" pod "6def2200-c6a3-41b1-b528-959c54018a4d" (UID: "6def2200-c6a3-41b1-b528-959c54018a4d"). InnerVolumeSpecName "kube-api-access-z9n4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:17.011805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.011786 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-metrics-certs" pod "6def2200-c6a3-41b1-b528-959c54018a4d" (UID: "6def2200-c6a3-41b1-b528-959c54018a4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:17.080699 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.080653 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:17.112921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.112881 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.112921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.112912 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9n4b\" (UniqueName: \"kubernetes.io/projected/6def2200-c6a3-41b1-b528-959c54018a4d-kube-api-access-z9n4b\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.112921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.112925 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6def2200-c6a3-41b1-b528-959c54018a4d-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.585357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.585321 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:17.631119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.631087 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" event={"ID":"6def2200-c6a3-41b1-b528-959c54018a4d","Type":"ContainerDied","Data":"4b71b6b6f86d6af334515ee0125075f5cf3629e56293ac23e6496e8a9def4798"} Oct 30 00:29:17.631270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.631154 4433 scope.go:122] "RemoveContainer" containerID="ee362b5d1990be71fc93fcfbdcd23af5c2ac002e8f6df02271f180df941c15f7" Oct 30 00:29:17.631270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.631094 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k" Oct 30 00:29:17.687065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.687037 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k"] Oct 30 00:29:17.699623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:17.699560 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-00df5f-596f55bc9c-nkd7k"] Oct 30 00:29:18.727114 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:18.727065 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:19.180538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.180505 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6def2200-c6a3-41b1-b528-959c54018a4d" path="/var/lib/kubelet/pods/6def2200-c6a3-41b1-b528-959c54018a4d/volumes" Oct 30 00:29:19.364875 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.364829 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96"] Oct 30 00:29:19.365873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365851 4433 cpu_manager.go:414] "containerMap: removing container" podUID="580149e6-952f-43dc-84cd-4e004513bbbc" containerName="payload-processing" Oct 30 00:29:19.365873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365872 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580149e6-952f-43dc-84cd-4e004513bbbc" containerName="payload-processing" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365895 4433 cpu_manager.go:414] "containerMap: removing container" podUID="6def2200-c6a3-41b1-b528-959c54018a4d" containerName="maas-api" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365902 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6def2200-c6a3-41b1-b528-959c54018a4d" containerName="maas-api" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365930 4433 cpu_manager.go:414] "containerMap: removing container" podUID="63bebbfd-3ad5-423f-8653-0d257c05dbef" containerName="payload-processing" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365938 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bebbfd-3ad5-423f-8653-0d257c05dbef" containerName="payload-processing" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365987 4433 cpu_manager.go:414] "containerMap: removing container" podUID="424d4f13-177e-4ef4-986d-60f788399800" containerName="payload-pre-processing" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.365995 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424d4f13-177e-4ef4-986d-60f788399800" containerName="payload-pre-processing" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366008 4433 cpu_manager.go:414] "containerMap: removing container" podUID="e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" containerName="maas-api" Oct 30 00:29:19.366024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366015 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366046 4433 cpu_manager.go:414] "containerMap: removing container" podUID="bccd33e5-4ecd-4528-b879-a460841f4e90" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366054 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccd33e5-4ecd-4528-b879-a460841f4e90" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366066 4433 cpu_manager.go:414] "containerMap: removing container" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366073 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366087 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366095 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366106 4433 cpu_manager.go:414] "containerMap: removing container" podUID="cd0df4b3-3098-41d8-84ca-23c5a23b3183" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366113 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0df4b3-3098-41d8-84ca-23c5a23b3183" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366126 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1b98754c-b660-4dbe-aa39-a01a1fb1fe16" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366131 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b98754c-b660-4dbe-aa39-a01a1fb1fe16" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366140 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d51c7cd0-80c4-48cb-a6be-67054065240c" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366145 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d51c7cd0-80c4-48cb-a6be-67054065240c" containerName="maas-api" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366153 4433 cpu_manager.go:414] "containerMap: removing container" podUID="8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" containerName="payload-pre-processing" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366157 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" containerName="payload-pre-processing" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366172 4433 cpu_manager.go:414] "containerMap: removing container" podUID="867d9d08-9c17-422a-8643-3127b4aa8589" containerName="payload-pre-processing" Oct 30 00:29:19.366293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366177 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867d9d08-9c17-422a-8643-3127b4aa8589" containerName="payload-pre-processing" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366317 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bebbfd-3ad5-423f-8653-0d257c05dbef" containerName="payload-processing" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366328 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="424d4f13-177e-4ef4-986d-60f788399800" containerName="payload-pre-processing" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366334 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cd303d-ba4a-4dd1-a4fa-442846dff3d7" containerName="maas-api" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366344 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccd33e5-4ecd-4528-b879-a460841f4e90" containerName="maas-api" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366349 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="306325e3-eb7b-4199-93f1-27016cd6e3ff" containerName="istio-proxy" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366356 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d148256-ff4d-48eb-b0a6-1c28033d6b51" containerName="istio-proxy" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366360 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0df4b3-3098-41d8-84ca-23c5a23b3183" containerName="maas-api" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366368 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b98754c-b660-4dbe-aa39-a01a1fb1fe16" containerName="maas-api" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366374 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d51c7cd0-80c4-48cb-a6be-67054065240c" containerName="maas-api" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366380 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4e6491-6a6c-4e8f-a9d7-ac2b59163c2f" containerName="payload-pre-processing" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366387 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="867d9d08-9c17-422a-8643-3127b4aa8589" containerName="payload-pre-processing" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366394 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="580149e6-952f-43dc-84cd-4e004513bbbc" containerName="payload-processing" Oct 30 00:29:19.366858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.366401 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6def2200-c6a3-41b1-b528-959c54018a4d" containerName="maas-api" Oct 30 00:29:19.371635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.371614 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.393543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.393513 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96"] Oct 30 00:29:19.400743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.400709 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-93016b-openshift-default-dockercfg-6ddrc\"" Oct 30 00:29:19.448965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.448893 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0152a495-2048-4d28-8aa4-2bde079180e2-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.448965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.448940 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.449171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.449032 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.449171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.449153 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0152a495-2048-4d28-8aa4-2bde079180e2-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.449266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.449239 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.449349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.449333 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.449440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.449426 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.449544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.449532 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.449654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.449557 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78m4\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-kube-api-access-l78m4\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.550661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.550631 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.550856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.550673 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.550856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.550741 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0152a495-2048-4d28-8aa4-2bde079180e2-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.550856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.550761 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.550856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.550819 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.550938 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551117 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551193 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551159 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551159 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78m4\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-kube-api-access-l78m4\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551228 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551254 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0152a495-2048-4d28-8aa4-2bde079180e2-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551324 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551598 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.551904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.551881 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0152a495-2048-4d28-8aa4-2bde079180e2-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.553272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.553255 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0152a495-2048-4d28-8aa4-2bde079180e2-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.553272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.553263 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.574637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.574607 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.574805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.574788 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78m4\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-kube-api-access-l78m4\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:19.687404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:19.687366 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:20.015400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.015370 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96"] Oct 30 00:29:20.015834 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:29:20.015559 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0152a495_2048_4d28_8aa4_2bde079180e2.slice/crio-b9aa57edf3a5afe64089cd3c3c45cf498bb62ab106d2b150e92af6a18b84a838 WatchSource:0}: Error finding container b9aa57edf3a5afe64089cd3c3c45cf498bb62ab106d2b150e92af6a18b84a838: Status 404 returned error can't find the container with id b9aa57edf3a5afe64089cd3c3c45cf498bb62ab106d2b150e92af6a18b84a838 Oct 30 00:29:20.018424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.018392 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:29:20.018545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.018457 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:29:20.018545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.018509 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:29:20.088352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.088320 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:20.654976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.654937 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" event={"ID":"0152a495-2048-4d28-8aa4-2bde079180e2","Type":"ContainerStarted","Data":"587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70"} Oct 30 00:29:20.655159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.655001 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" event={"ID":"0152a495-2048-4d28-8aa4-2bde079180e2","Type":"ContainerStarted","Data":"b9aa57edf3a5afe64089cd3c3c45cf498bb62ab106d2b150e92af6a18b84a838"} Oct 30 00:29:20.688187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.688148 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:20.729949 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:20.729898 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podStartSLOduration=1.729885043 podStartE2EDuration="1.729885043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:20.713850012 +0000 UTC m=+1469.647918560" watchObservedRunningTime="2026-10-30 00:29:20.729885043 +0000 UTC m=+1469.663953639" Oct 30 00:29:21.579593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:21.578459 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:21.695452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:21.695409 4433 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:29:21.695677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:21.695477 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 00:29:22.691121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:22.691083 4433 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:29:22.691636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:22.691152 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 00:29:22.732127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:22.732101 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:22.920462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:22.920425 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:23.019694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:23.019594 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:23.283652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:23.269980 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:23.383356 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:23.383317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:23.491598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:23.491076 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:23.690596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:23.690050 4433 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:29:23.690596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:23.690127 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 00:29:23.900977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:23.900942 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:24.604768 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:24.604735 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:24.695157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:24.694022 4433 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:29:24.695157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:24.694087 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 00:29:24.707156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:24.705776 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:24.999721 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:24.999618 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:25.302938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:25.302828 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:25.600384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:25.600352 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x"] Oct 30 00:29:25.692648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:25.690801 4433 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:29:25.692648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:25.690856 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 00:29:25.692648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:25.691645 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:25.806773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:25.801769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:26.688175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:26.688131 4433 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:29:26.688732 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:26.688219 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 00:29:27.422940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:27.422893 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:27.692394 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:27.692314 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:27.692806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:27.692754 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:27.693467 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:27.693446 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:29:29.310975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:29.310942 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:30.316373 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:30.316342 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:30.417332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:30.417286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:30.601585 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:30.601487 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" podUID="9a9e143b-bb92-4c5d-bcb1-470341b19946" containerName="payload-pre-processing" containerID="cri-o://79b209d09bf9c13f39ff234c5b8df01dbaaeeac8063cbddf09296862737eddb8" gracePeriod=25 Oct 30 00:29:30.691116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:30.691065 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:30.843093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:30.842617 4433 generic.go:356] "Generic (PLEG): container finished" podID="9a9e143b-bb92-4c5d-bcb1-470341b19946" containerID="79b209d09bf9c13f39ff234c5b8df01dbaaeeac8063cbddf09296862737eddb8" exitCode=0 Oct 30 00:29:30.843093 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:30.842672 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" event={"ID":"9a9e143b-bb92-4c5d-bcb1-470341b19946","Type":"ContainerDied","Data":"79b209d09bf9c13f39ff234c5b8df01dbaaeeac8063cbddf09296862737eddb8"} Oct 30 00:29:30.995635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:30.993595 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:29:31.097622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.095535 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:31.146985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.146941 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9e143b-bb92-4c5d-bcb1-470341b19946-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9e143b-bb92-4c5d-bcb1-470341b19946-plugins-config-volume\") pod \"9a9e143b-bb92-4c5d-bcb1-470341b19946\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " Oct 30 00:29:31.147178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.147160 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9e143b-bb92-4c5d-bcb1-470341b19946-kube-api-access-lpbsl\" (UniqueName: \"kubernetes.io/projected/9a9e143b-bb92-4c5d-bcb1-470341b19946-kube-api-access-lpbsl\") pod \"9a9e143b-bb92-4c5d-bcb1-470341b19946\" (UID: \"9a9e143b-bb92-4c5d-bcb1-470341b19946\") " Oct 30 00:29:31.147677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.147583 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9e143b-bb92-4c5d-bcb1-470341b19946-plugins-config-volume" pod "9a9e143b-bb92-4c5d-bcb1-470341b19946" (UID: "9a9e143b-bb92-4c5d-bcb1-470341b19946"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:31.148316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.148289 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9e143b-bb92-4c5d-bcb1-470341b19946-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:31.150667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.150630 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9e143b-bb92-4c5d-bcb1-470341b19946-kube-api-access-lpbsl" pod "9a9e143b-bb92-4c5d-bcb1-470341b19946" (UID: "9a9e143b-bb92-4c5d-bcb1-470341b19946"). InnerVolumeSpecName "kube-api-access-lpbsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:31.250349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.250311 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpbsl\" (UniqueName: \"kubernetes.io/projected/9a9e143b-bb92-4c5d-bcb1-470341b19946-kube-api-access-lpbsl\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:31.851057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.851028 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" Oct 30 00:29:31.851057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.851049 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x" event={"ID":"9a9e143b-bb92-4c5d-bcb1-470341b19946","Type":"ContainerDied","Data":"bdf25b689c49b0b3163159a5a0cb62308594d70dc7c556de62e0710fb8305dd7"} Oct 30 00:29:31.851549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.851112 4433 scope.go:122] "RemoveContainer" containerID="79b209d09bf9c13f39ff234c5b8df01dbaaeeac8063cbddf09296862737eddb8" Oct 30 00:29:31.905860 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.905822 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x"] Oct 30 00:29:31.918691 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:31.918665 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-20d942-5f67b4c-chq2x"] Oct 30 00:29:32.334289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:32.334259 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:32.501319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:32.501287 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:32.608158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:32.608076 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:32.817592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:32.812600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:32.923159 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:32.923050 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:33.024803 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:33.024764 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:33.181122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:33.181043 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9e143b-bb92-4c5d-bcb1-470341b19946" path="/var/lib/kubelet/pods/9a9e143b-bb92-4c5d-bcb1-470341b19946/volumes" Oct 30 00:29:33.463331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:33.463245 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:34.560465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:34.560429 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:36.004715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:36.004678 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:37.101341 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:29:37.101307 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:29:37.354271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:37.354190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:38.709022 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:38.708988 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:38.822496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:38.822456 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:39.804232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:39.804190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:39.914674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:39.914639 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:39.949559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:39.949519 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km"] Oct 30 00:29:39.949989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:39.949841 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" podUID="77773510-35f2-4b2c-a884-a381ca014eeb" containerName="maas-api" containerID="cri-o://a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2" gracePeriod=30 Oct 30 00:29:40.026423 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.026391 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:40.152729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.152686 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:40.250912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.250884 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:29:40.313670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.311350 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77773510-35f2-4b2c-a884-a381ca014eeb-kube-api-access-v6qtn\" (UniqueName: \"kubernetes.io/projected/77773510-35f2-4b2c-a884-a381ca014eeb-kube-api-access-v6qtn\") pod \"77773510-35f2-4b2c-a884-a381ca014eeb\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " Oct 30 00:29:40.313670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.311442 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls\") pod \"77773510-35f2-4b2c-a884-a381ca014eeb\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " Oct 30 00:29:40.313670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.311469 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77773510-35f2-4b2c-a884-a381ca014eeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77773510-35f2-4b2c-a884-a381ca014eeb-openshift-service-ca\") pod \"77773510-35f2-4b2c-a884-a381ca014eeb\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " Oct 30 00:29:40.313670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.311869 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-metrics-certs\") pod \"77773510-35f2-4b2c-a884-a381ca014eeb\" (UID: \"77773510-35f2-4b2c-a884-a381ca014eeb\") " Oct 30 00:29:40.316592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.316542 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77773510-35f2-4b2c-a884-a381ca014eeb-kube-api-access-v6qtn" pod "77773510-35f2-4b2c-a884-a381ca014eeb" (UID: "77773510-35f2-4b2c-a884-a381ca014eeb"). InnerVolumeSpecName "kube-api-access-v6qtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:40.328057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.327766 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls" pod "77773510-35f2-4b2c-a884-a381ca014eeb" (UID: "77773510-35f2-4b2c-a884-a381ca014eeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:40.328670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.328354 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77773510-35f2-4b2c-a884-a381ca014eeb-openshift-service-ca" pod "77773510-35f2-4b2c-a884-a381ca014eeb" (UID: "77773510-35f2-4b2c-a884-a381ca014eeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:40.340065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.340029 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-metrics-certs" pod "77773510-35f2-4b2c-a884-a381ca014eeb" (UID: "77773510-35f2-4b2c-a884-a381ca014eeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:40.420759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.420727 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:40.420995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.420978 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6qtn\" (UniqueName: \"kubernetes.io/projected/77773510-35f2-4b2c-a884-a381ca014eeb-kube-api-access-v6qtn\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:40.421097 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.421083 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77773510-35f2-4b2c-a884-a381ca014eeb-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:40.421179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.421168 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77773510-35f2-4b2c-a884-a381ca014eeb-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:29:40.487554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.487521 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:40.593331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.593234 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:40.700147 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.700112 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:40.931010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.930921 4433 generic.go:356] "Generic (PLEG): container finished" podID="77773510-35f2-4b2c-a884-a381ca014eeb" containerID="a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2" exitCode=0 Oct 30 00:29:40.931010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.930978 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" Oct 30 00:29:40.931523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.931004 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" event={"ID":"77773510-35f2-4b2c-a884-a381ca014eeb","Type":"ContainerDied","Data":"a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2"} Oct 30 00:29:40.931523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.931065 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km" event={"ID":"77773510-35f2-4b2c-a884-a381ca014eeb","Type":"ContainerDied","Data":"c26395052a25cc463775aa592d9b65a5a039451253f65aa82f26566b7cb2237c"} Oct 30 00:29:40.931523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.931096 4433 scope.go:122] "RemoveContainer" containerID="a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2" Oct 30 00:29:40.943485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.943450 4433 scope.go:122] "RemoveContainer" containerID="a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2" Oct 30 00:29:40.943853 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:29:40.943823 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2\": container with ID starting with a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2 not found: ID does not exist" containerID="a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2" Oct 30 00:29:40.943962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.943866 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2"} err="failed to get container status \"a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2\": rpc error: code = NotFound desc = could not find container \"a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2\": container with ID starting with a239fa4574da3f652be19410573d6974d9941421e350346b39aa45ba15c6def2 not found: ID does not exist" Oct 30 00:29:40.986607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.986552 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km"] Oct 30 00:29:40.999437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:40.999401 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-20d942-5c8c5cffd7-ss6km"] Oct 30 00:29:41.110106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:41.110075 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:41.182248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:41.182175 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77773510-35f2-4b2c-a884-a381ca014eeb" path="/var/lib/kubelet/pods/77773510-35f2-4b2c-a884-a381ca014eeb/volumes" Oct 30 00:29:42.121342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:42.121308 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:43.574326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:43.574293 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:44.982887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:44.982799 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:46.885027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:46.884998 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:47.230444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:47.230338 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:47.335889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:47.335853 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:47.604424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:47.604390 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:48.861137 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:48.861098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:49.615955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:49.615919 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:49.724561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:49.724523 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:50.473655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:50.473615 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:52.225597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:52.223524 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:53.046812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:53.046762 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:29:53.104608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:53.102834 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:53.140806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:53.140704 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:53.156061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:53.155606 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:54.174998 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:54.174956 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:54.280132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:54.280102 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:55.411880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:55.411846 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:29:55.418601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:55.418564 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:29:55.775775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:55.775684 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:57.103067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:57.103024 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:58.533798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:58.533763 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:58.723554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:58.723527 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log" Oct 30 00:29:58.937067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:58.937039 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log" Oct 30 00:29:59.464497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:59.464464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:29:59.703741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:29:59.703707 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:00.465383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:00.465348 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:00.568349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:00.568315 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:01.096275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:01.096239 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:02.048648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:02.048562 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:03.512074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:03.512035 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:05.086339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:05.086301 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:07.101269 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:30:07.101235 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:30:07.182229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:07.182190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:07.296004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:07.295971 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:07.758952 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:07.758912 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:09.160441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:09.160408 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:09.497679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:09.497548 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:10.027966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:10.027924 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:10.256617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:10.256541 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:10.632322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:10.632282 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:12.076209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.076172 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:12.213851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.213818 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv"] Oct 30 00:30:12.214613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.214559 4433 cpu_manager.go:414] "containerMap: removing container" podUID="77773510-35f2-4b2c-a884-a381ca014eeb" containerName="maas-api" Oct 30 00:30:12.214613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.214595 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77773510-35f2-4b2c-a884-a381ca014eeb" containerName="maas-api" Oct 30 00:30:12.214795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.214645 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9a9e143b-bb92-4c5d-bcb1-470341b19946" containerName="payload-pre-processing" Oct 30 00:30:12.214795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.214651 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9e143b-bb92-4c5d-bcb1-470341b19946" containerName="payload-pre-processing" Oct 30 00:30:12.214867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.214793 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="77773510-35f2-4b2c-a884-a381ca014eeb" containerName="maas-api" Oct 30 00:30:12.214867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.214814 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9e143b-bb92-4c5d-bcb1-470341b19946" containerName="payload-pre-processing" Oct 30 00:30:12.220233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.220212 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.227361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.227333 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf"] Oct 30 00:30:12.248682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.247291 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.259035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.258992 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.259273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.259251 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.259492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.259475 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.259649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.259632 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6r6l\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-kube-api-access-v6r6l\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.260160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.260140 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.260357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.260323 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.260556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.260538 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.260759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.260738 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f310099f-ae88-4cfe-9ee1-3175b696ab12-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.260958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.260937 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.277539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.277508 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv"] Oct 30 00:30:12.283670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.283638 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf"] Oct 30 00:30:12.363185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363105 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b75034-6711-4f76-9105-32372500328c-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363141 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4h7\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-kube-api-access-wk4h7\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363160 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b75034-6711-4f76-9105-32372500328c-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363287 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363333 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363425 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363376 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363435 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363468 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363525 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363508 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363531 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363547 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f310099f-ae88-4cfe-9ee1-3175b696ab12-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363564 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363732 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363757 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.363832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363804 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363861 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.363999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.363973 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.364170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.364059 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.364170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.364092 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6r6l\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-kube-api-access-v6r6l\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.364170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.364106 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.364170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.364152 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.364170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.364163 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.364452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.364381 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f310099f-ae88-4cfe-9ee1-3175b696ab12-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.365884 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.365862 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.366075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.366055 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.388208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.388174 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.388466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.388441 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6r6l\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-kube-api-access-v6r6l\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.465545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465497 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b75034-6711-4f76-9105-32372500328c-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.465545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465548 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4h7\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-kube-api-access-wk4h7\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.465842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465583 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b75034-6711-4f76-9105-32372500328c-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.465842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465626 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.465842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465648 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.465842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465693 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.465842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465830 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.466113 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.465905 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.466113 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.466048 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.466113 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.466095 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.466290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.466198 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.466351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.466289 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.466448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.466414 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b75034-6711-4f76-9105-32372500328c-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.466546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.466526 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.468237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.468217 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.468345 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.468328 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b75034-6711-4f76-9105-32372500328c-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.489822 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.489797 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.490042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.490018 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4h7\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-kube-api-access-wk4h7\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.547639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.547606 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:12.583950 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.583920 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:12.878236 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.878207 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv"] Oct 30 00:30:12.879020 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:30:12.878995 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf310099f_ae88_4cfe_9ee1_3175b696ab12.slice/crio-2195d4b3182f7a11adc63ae171e67824d710dfa85d0c03b56824d7b4d1a2ff8b WatchSource:0}: Error finding container 2195d4b3182f7a11adc63ae171e67824d710dfa85d0c03b56824d7b4d1a2ff8b: Status 404 returned error can't find the container with id 2195d4b3182f7a11adc63ae171e67824d710dfa85d0c03b56824d7b4d1a2ff8b Oct 30 00:30:12.881606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.881549 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:30:12.881687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.881632 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:30:12.881687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.881670 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:30:12.923270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.923242 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf"] Oct 30 00:30:12.924928 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:30:12.924902 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b75034_6711_4f76_9105_32372500328c.slice/crio-0d9a890fcdb8ecfa0301547b73ca003f57cbf952d2013cdc607a23a20c30b0d8 WatchSource:0}: Error finding container 0d9a890fcdb8ecfa0301547b73ca003f57cbf952d2013cdc607a23a20c30b0d8: Status 404 returned error can't find the container with id 0d9a890fcdb8ecfa0301547b73ca003f57cbf952d2013cdc607a23a20c30b0d8 Oct 30 00:30:12.927913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.927871 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:30:12.928013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.927961 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:30:12.928013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:12.928000 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:30:13.309995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.309962 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" event={"ID":"78b75034-6711-4f76-9105-32372500328c","Type":"ContainerStarted","Data":"055c0a9252aa64232e83a07b83dfbf0bfa1d76ebfb07d9182c213135e2e3f0e4"} Oct 30 00:30:13.310493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.310026 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" event={"ID":"78b75034-6711-4f76-9105-32372500328c","Type":"ContainerStarted","Data":"0d9a890fcdb8ecfa0301547b73ca003f57cbf952d2013cdc607a23a20c30b0d8"} Oct 30 00:30:13.312135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.312105 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" event={"ID":"f310099f-ae88-4cfe-9ee1-3175b696ab12","Type":"ContainerStarted","Data":"70dc0174219e9d532565fd16c872c784c826defaef469901ca4a7b426605ee16"} Oct 30 00:30:13.312268 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.312188 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" event={"ID":"f310099f-ae88-4cfe-9ee1-3175b696ab12","Type":"ContainerStarted","Data":"2195d4b3182f7a11adc63ae171e67824d710dfa85d0c03b56824d7b4d1a2ff8b"} Oct 30 00:30:13.358264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.358188 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podStartSLOduration=1.358109104 podStartE2EDuration="1.358109104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:13.355340268 +0000 UTC m=+1522.289408825" watchObservedRunningTime="2026-10-30 00:30:13.358109104 +0000 UTC m=+1522.292177641" Oct 30 00:30:13.407725 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.407669 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podStartSLOduration=1.4076527589999999 podStartE2EDuration="1.407652759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:13.40696792 +0000 UTC m=+1522.341036458" watchObservedRunningTime="2026-10-30 00:30:13.407652759 +0000 UTC m=+1522.341721288" Oct 30 00:30:13.548774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.548739 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:13.551296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.551262 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:30:13.551460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.551326 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:30:13.584252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.584164 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:13.585779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.585743 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:13.585893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.585805 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:13.669510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:13.669461 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:14.095360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:14.094214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:14.210239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:14.210186 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:14.548940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:14.548866 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:30:14.548940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:14.548909 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:30:14.590596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:14.589969 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:14.590596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:14.590030 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:15.341658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:15.341621 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:15.548191 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:15.548156 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:30:15.548364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:15.548246 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:30:15.592648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:15.592515 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:15.592648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:15.592593 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:15.892648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:15.892466 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:16.004520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:16.004419 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:16.548943 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:16.548899 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:30:16.549131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:16.548975 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:30:16.599408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:16.599371 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:16.613656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:16.599995 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:17.202220 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:17.202184 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:17.548375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:17.548294 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:30:17.548532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:17.548360 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:30:17.598477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:17.597677 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:17.598477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:17.597740 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:18.548133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:18.548097 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:30:18.548618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:18.548563 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:30:18.586256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:18.586216 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:18.586549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:18.586517 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:19.121194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:19.121017 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:19.548374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:19.548286 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:30:19.548374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:19.548348 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:30:19.585286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:19.585230 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:19.585485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:19.585319 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:20.585067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:20.585025 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:30:20.585499 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:20.585096 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 00:30:20.692457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:20.692419 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:20.958932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:20.958834 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:20.959533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:20.959507 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:20.960898 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:20.960876 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:30:21.589481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:21.589442 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:21.590104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:21.589870 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:21.590668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:21.590647 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:30:22.265382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:22.265339 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:24.498727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.498673 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 00:30:24.510841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.510799 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.533286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.533240 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.600602 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.599799 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.631315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.631265 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.674601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.670603 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:24.779442 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.779340 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.817606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.809887 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.909258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.908676 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.939937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:24.914654 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.055130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.051229 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.055130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.053450 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.160604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.150954 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.160604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.151301 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:30:25.585643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.574451 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.619601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.586731 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:30:25.619895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.619861 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.649735 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.621445 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:30:25.664760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.664718 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:30:25.666372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:25.666346 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:30:26.054287 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:26.053909 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:26.060122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:26.054589 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:30:26.079917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:26.078089 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:30:26.094519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:26.093514 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:30:26.099512 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:26.098676 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:30:26.155204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:26.155173 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:30:26.611948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:26.611908 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:28.480435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:28.480401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:30.129545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:30.129509 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:31.504303 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:31.504223 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:32.965112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:32.965081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:33.073856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:33.073815 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:33.289038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:33.288953 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:34.491107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:34.491054 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:36.005706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:36.005658 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:37.110052 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:30:37.110011 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:30:37.591109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:37.591072 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:38.829883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:38.829845 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:39.132235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:39.132157 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:39.290801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:39.290739 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:39.414133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:39.414015 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:43.346810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.346776 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw"] Oct 30 00:30:43.347454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.347042 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" podUID="d70e5015-960b-40b6-a828-b18393dfc9ad" containerName="payload-pre-processing" containerID="cri-o://4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b" gracePeriod=30 Oct 30 00:30:43.395428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.395389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:43.517965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.517927 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:43.784607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.779614 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:43.791443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.791404 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:30:43.893713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.893678 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70e5015-960b-40b6-a828-b18393dfc9ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70e5015-960b-40b6-a828-b18393dfc9ad-plugins-config-volume\") pod \"d70e5015-960b-40b6-a828-b18393dfc9ad\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " Oct 30 00:30:43.894073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.893956 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70e5015-960b-40b6-a828-b18393dfc9ad-kube-api-access-rmkgq\" (UniqueName: \"kubernetes.io/projected/d70e5015-960b-40b6-a828-b18393dfc9ad-kube-api-access-rmkgq\") pod \"d70e5015-960b-40b6-a828-b18393dfc9ad\" (UID: \"d70e5015-960b-40b6-a828-b18393dfc9ad\") " Oct 30 00:30:43.895736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.895671 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70e5015-960b-40b6-a828-b18393dfc9ad-plugins-config-volume" pod "d70e5015-960b-40b6-a828-b18393dfc9ad" (UID: "d70e5015-960b-40b6-a828-b18393dfc9ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:43.898942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.898881 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:43.899445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.899425 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70e5015-960b-40b6-a828-b18393dfc9ad-kube-api-access-rmkgq" pod "d70e5015-960b-40b6-a828-b18393dfc9ad" (UID: "d70e5015-960b-40b6-a828-b18393dfc9ad"). InnerVolumeSpecName "kube-api-access-rmkgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:43.996550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.996475 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmkgq\" (UniqueName: \"kubernetes.io/projected/d70e5015-960b-40b6-a828-b18393dfc9ad-kube-api-access-rmkgq\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:30:43.996550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:43.996505 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70e5015-960b-40b6-a828-b18393dfc9ad-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:30:44.004418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.004384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:44.263759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.263669 4433 generic.go:356] "Generic (PLEG): container finished" podID="d70e5015-960b-40b6-a828-b18393dfc9ad" containerID="4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b" exitCode=0 Oct 30 00:30:44.263759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.263724 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" Oct 30 00:30:44.263983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.263754 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" event={"ID":"d70e5015-960b-40b6-a828-b18393dfc9ad","Type":"ContainerDied","Data":"4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b"} Oct 30 00:30:44.263983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.263809 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw" event={"ID":"d70e5015-960b-40b6-a828-b18393dfc9ad","Type":"ContainerDied","Data":"603de2ed9b8a99e7b998244d1b451fd79305017a4e19dd389e993720eca34fb2"} Oct 30 00:30:44.263983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.263831 4433 scope.go:122] "RemoveContainer" containerID="4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b" Oct 30 00:30:44.275534 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.275512 4433 scope.go:122] "RemoveContainer" containerID="4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b" Oct 30 00:30:44.275876 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:30:44.275856 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b\": container with ID starting with 4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b not found: ID does not exist" containerID="4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b" Oct 30 00:30:44.275957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.275888 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b"} err="failed to get container status \"4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b\": rpc error: code = NotFound desc = could not find container \"4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b\": container with ID starting with 4d5653872f3ea04cc7cb63f9b34094b144f2ecf3672afc06a0adc28bdbb5fd8b not found: ID does not exist" Oct 30 00:30:44.318637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.318603 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw"] Oct 30 00:30:44.323124 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.323095 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-557f888647-xjjtw"] Oct 30 00:30:44.438255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:44.438216 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:45.181441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:45.181408 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70e5015-960b-40b6-a828-b18393dfc9ad" path="/var/lib/kubelet/pods/d70e5015-960b-40b6-a828-b18393dfc9ad/volumes" Oct 30 00:30:45.845652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:45.845583 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:47.383330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:47.383293 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:48.965733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:48.965697 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:49.466259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:49.466219 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:50.354830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:50.354799 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:52.077683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:52.077631 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:53.568514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:53.568477 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:54.796440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:54.796350 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:54.954104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:54.954069 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:56.045291 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:56.045254 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:56.517837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:56.511298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:56.623696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:56.623657 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:56.751026 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:56.750990 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:57.941996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:57.941959 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:59.215675 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:59.215640 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:59.780075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:59.780020 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:30:59.893627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:30:59.893544 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:00.003496 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:00.003464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:00.411047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:00.411010 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:00.530267 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:00.530224 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:00.647509 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:00.647473 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:01.384670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:01.384631 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:02.145392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:02.145353 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:02.253421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:02.253384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:03.373547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:03.373509 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:04.429979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:04.429937 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:04.797196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:04.796804 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:07.008479 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:07.008445 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:07.105682 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:31:07.105640 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:31:07.694668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:07.694634 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:09.097452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.097415 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:09.445772 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.445697 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f"] Oct 30 00:31:09.446816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.446792 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d70e5015-960b-40b6-a828-b18393dfc9ad" containerName="payload-pre-processing" Oct 30 00:31:09.446872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.446821 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70e5015-960b-40b6-a828-b18393dfc9ad" containerName="payload-pre-processing" Oct 30 00:31:09.447126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.447109 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70e5015-960b-40b6-a828-b18393dfc9ad" containerName="payload-pre-processing" Oct 30 00:31:09.451383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.451360 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.473229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.473197 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f"] Oct 30 00:31:09.537051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537022 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537060 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgj58\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-kube-api-access-pgj58\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537085 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537299 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537255 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537299 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537290 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537380 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537458 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537436 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537481 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.537655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.537607 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639249 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639374 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639400 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgj58\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-kube-api-access-pgj58\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639428 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639534 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639586 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639677 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639678 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639705 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639876 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639747 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.639975 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.639950 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.640060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.640042 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.640139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.640115 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.640340 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.640323 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.641684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.641663 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.642053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.642033 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.667753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.667729 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgj58\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-kube-api-access-pgj58\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.667882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.667801 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:09.774638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:09.774517 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:10.100523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.100488 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f"] Oct 30 00:31:10.104020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.103969 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:31:10.104138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.104055 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:31:10.104138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.104096 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:31:10.666306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.666274 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:10.676372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.676338 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" event={"ID":"a7dcedc9-87ca-44ac-8ce2-2267892d50a0","Type":"ContainerStarted","Data":"1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14"} Oct 30 00:31:10.676530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.676394 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" event={"ID":"a7dcedc9-87ca-44ac-8ce2-2267892d50a0","Type":"ContainerStarted","Data":"f855f0e0812ceba8afd3b6e649bf6a464e8ccf98e5070639a0c3b59c4f333ea3"} Oct 30 00:31:10.739595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.735896 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podStartSLOduration=1.7358734839999999 podStartE2EDuration="1.735873484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:10.734244839 +0000 UTC m=+1579.668313396" watchObservedRunningTime="2026-10-30 00:31:10.735873484 +0000 UTC m=+1579.669942024" Oct 30 00:31:10.775678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:10.775644 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:11.779127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:11.779093 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:31:11.779636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:11.779147 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:31:12.190121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:12.190089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:12.776932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:12.776887 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:31:12.777150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:12.776947 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:31:13.778604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:13.777724 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:13.778604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:13.778267 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:31:13.778604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:13.778329 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:31:14.778183 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:14.778146 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:31:14.778524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:14.778493 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:31:15.130881 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:15.130841 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:15.775328 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:15.775290 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:31:15.775639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:15.775607 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:31:15.923730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:15.923696 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:16.726293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:16.726255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:16.775565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:16.775525 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:31:16.775749 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:16.775624 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:31:16.835757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:16.835716 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:16.944326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:16.944281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:17.781275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:17.781241 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:17.781771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:17.781752 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:17.782550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:17.782530 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:31:18.380275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:18.380242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:19.891214 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:19.891181 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:21.897076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:21.897040 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:23.371043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:23.371003 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:24.764095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:24.764052 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:25.809844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:25.809809 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:26.220520 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:26.220411 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:27.433731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:27.433675 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:27.693524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:27.693431 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:27.816752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:27.816561 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:27.922868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:27.922829 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:29.044565 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:29.044520 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:30.584454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:30.584414 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:32.082760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:32.082718 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:33.495943 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:33.495906 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:33.607048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:33.607015 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:33.902386 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:33.902345 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj"] Oct 30 00:31:33.902617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:33.902590 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" containerID="cri-o://2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd" gracePeriod=30 Oct 30 00:31:33.959014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:33.958981 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:34.276530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:34.276426 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:34.383296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:34.383257 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:35.192789 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:35.192744 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:36.718505 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:36.718467 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:37.128803 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:31:37.128749 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:31:37.130748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.130723 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g"] Oct 30 00:31:37.135590 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.135550 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.161989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.161947 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g"] Oct 30 00:31:37.333293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333257 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333306 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333399 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swg2\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-kube-api-access-9swg2\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333514 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333665 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333639 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333680 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333818 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333852 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.333954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.333942 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435278 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435323 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435349 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435387 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swg2\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-kube-api-access-9swg2\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435495 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435586 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435618 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435734 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435763 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435787 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435816 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.435976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.435900 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.436102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.436087 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.436385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.436366 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.438056 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.438032 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.438157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.438033 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.459302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.459278 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.459413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.459395 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swg2\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-kube-api-access-9swg2\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:37.752606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:37.752498 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:38.083799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.083773 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g"] Oct 30 00:31:38.086001 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:31:38.085969 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade4e4dc_fc17_4e27_901f_9321d6ea55e4.slice/crio-5287c7c4de876762efef4527d6d53bbd9e40a4379775ba2375983221cdef352e WatchSource:0}: Error finding container 5287c7c4de876762efef4527d6d53bbd9e40a4379775ba2375983221cdef352e: Status 404 returned error can't find the container with id 5287c7c4de876762efef4527d6d53bbd9e40a4379775ba2375983221cdef352e Oct 30 00:31:38.088980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.088943 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:31:38.089071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.089009 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:31:38.089071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.089038 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:31:38.403851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.403760 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:38.958071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.958035 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:38.962926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.962888 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" event={"ID":"ade4e4dc-fc17-4e27-901f-9321d6ea55e4","Type":"ContainerStarted","Data":"f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54"} Oct 30 00:31:38.963115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:38.962956 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" event={"ID":"ade4e4dc-fc17-4e27-901f-9321d6ea55e4","Type":"ContainerStarted","Data":"5287c7c4de876762efef4527d6d53bbd9e40a4379775ba2375983221cdef352e"} Oct 30 00:31:39.023507 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.023457 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podStartSLOduration=2.023442737 podStartE2EDuration="2.023442737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:39.021453495 +0000 UTC m=+1607.955522040" watchObservedRunningTime="2026-10-30 00:31:39.023442737 +0000 UTC m=+1607.957511348" Oct 30 00:31:39.212686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.212659 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:31:39.214826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.214800 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:39.270085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.270048 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-certs\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.270271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.270226 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-socket\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.270411 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.270382 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-certs" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.270632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.270548 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-socket" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.270711 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.270647 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-kube-api-access-9h8jg\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-kube-api-access-9h8jg\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.271113 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.271088 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-podinfo\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.271437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.271415 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-token\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.271529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.271473 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-envoy\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.272174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.272139 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-data" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.272273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.271530 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-data\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.272400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.272382 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istiod-ca-cert\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.272741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.272714 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istiod-ca-cert" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:39.272839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.272784 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-credential-socket\") pod \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\" (UID: \"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc\") " Oct 30 00:31:39.273129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.273104 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-credential-socket" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.273720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.273697 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-kube-api-access-9h8jg" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "kube-api-access-9h8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:39.273963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.273931 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-podinfo" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:39.274742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.274315 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.274742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.274341 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h8jg\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-kube-api-access-9h8jg\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.274742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.274355 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.274742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.274364 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.274742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.274372 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.274742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.274380 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.274742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.274389 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.275954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.275931 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-token" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:39.276115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.276097 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-envoy" pod "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" (UID: "5ebc67f8-bd38-44fd-ab6d-42b50bd609fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.376324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.376282 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.376324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.376319 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.771598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.764420 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:39.771598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.764619 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:31:39.771598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.764671 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 00:31:39.986511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.986472 4433 generic.go:356] "Generic (PLEG): container finished" podID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerID="2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd" exitCode=0 Oct 30 00:31:39.987060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.986531 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" Oct 30 00:31:39.987060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.986552 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" event={"ID":"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc","Type":"ContainerDied","Data":"2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd"} Oct 30 00:31:39.987060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.986649 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj" event={"ID":"5ebc67f8-bd38-44fd-ab6d-42b50bd609fc","Type":"ContainerDied","Data":"1f5be3e8aaf50ee43f19e23854f90dede68108fc53f493f34f8cafdc47495fa8"} Oct 30 00:31:39.987060 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.986676 4433 scope.go:122] "RemoveContainer" containerID="2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd" Oct 30 00:31:39.999863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:39.999839 4433 scope.go:122] "RemoveContainer" containerID="0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40" Oct 30 00:31:40.012785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.012761 4433 scope.go:122] "RemoveContainer" containerID="2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd" Oct 30 00:31:40.013148 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:31:40.013125 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd\": container with ID starting with 2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd not found: ID does not exist" containerID="2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd" Oct 30 00:31:40.013217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.013158 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd"} err="failed to get container status \"2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd\": rpc error: code = NotFound desc = could not find container \"2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd\": container with ID starting with 2b792eaab341bf57d3e7f7ba193e6be1a4eaae70b635cdd0fdab367b5a2418fd not found: ID does not exist" Oct 30 00:31:40.013217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.013176 4433 scope.go:122] "RemoveContainer" containerID="0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40" Oct 30 00:31:40.013509 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:31:40.013476 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40\": container with ID starting with 0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40 not found: ID does not exist" containerID="0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40" Oct 30 00:31:40.013638 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.013514 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40"} err="failed to get container status \"0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40\": rpc error: code = NotFound desc = could not find container \"0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40\": container with ID starting with 0db6e3468bdc984a969bd4d790e934ddcf27dca1260002881e794caf291afc40 not found: ID does not exist" Oct 30 00:31:40.043607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.043471 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj"] Oct 30 00:31:40.082650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.080423 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-l2xnj"] Oct 30 00:31:40.408323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.408279 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:40.766164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.757957 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:31:40.766164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:40.758031 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 00:31:41.124953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:41.124918 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:41.228625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:41.228376 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" path="/var/lib/kubelet/pods/5ebc67f8-bd38-44fd-ab6d-42b50bd609fc/volumes" Oct 30 00:31:41.766598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:41.760007 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:31:41.766598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:41.760073 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 00:31:42.084439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:42.084404 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:42.203596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:42.203241 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:42.768601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:42.764974 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:31:42.768601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:42.765038 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 00:31:42.944595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:42.943944 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:43.773599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:43.764012 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:31:43.773599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:43.764074 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 00:31:43.880107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:43.880059 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:44.764598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:44.763955 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:31:44.764598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:44.764019 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 00:31:45.467694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:45.467664 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:45.753503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:45.753415 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:31:45.753503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:45.753485 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 00:31:46.757684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:46.757651 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:46.758086 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:46.758061 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:46.758683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:46.758661 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:31:46.988324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:46.988285 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:47.805476 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:47.805435 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m"] Oct 30 00:31:47.805925 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:47.805673 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" containerID="cri-o://806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351" gracePeriod=30 Oct 30 00:31:48.593522 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:48.593465 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:50.114595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:50.101270 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:51.732351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:51.731520 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:53.078295 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.078260 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:53.100532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.100466 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:31:53.256662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.256629 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-envoy\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.256826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.256695 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-podinfo\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.256826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.256815 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-token\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-token\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.256911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.256839 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-kube-api-access-pq6qw\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-kube-api-access-pq6qw\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.256911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.256859 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-certs\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.256911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.256892 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-socket\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.257072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257000 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-data\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.257131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257105 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05714c1e-3593-4b81-a683-fcba9d3438a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05714c1e-3593-4b81-a683-fcba9d3438a7-istiod-ca-cert\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.257182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257132 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-credential-socket\") pod \"05714c1e-3593-4b81-a683-fcba9d3438a7\" (UID: \"05714c1e-3593-4b81-a683-fcba9d3438a7\") " Oct 30 00:31:53.257182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257159 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-certs" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:53.257347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257321 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-socket" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:53.257472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257419 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-data" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:53.257542 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257518 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05714c1e-3593-4b81-a683-fcba9d3438a7-istiod-ca-cert" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:53.257700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257657 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-credential-socket" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:53.257996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257974 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.258080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.257998 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.258080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.258011 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.258080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.258021 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05714c1e-3593-4b81-a683-fcba9d3438a7-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.258080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.258029 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.259057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.259032 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-envoy" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:53.259154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.259144 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-token" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:53.259414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.259390 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-podinfo" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:53.259414 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.259406 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-kube-api-access-pq6qw" pod "05714c1e-3593-4b81-a683-fcba9d3438a7" (UID: "05714c1e-3593-4b81-a683-fcba9d3438a7"). InnerVolumeSpecName "kube-api-access-pq6qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:53.360163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.360084 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.360163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.360114 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.360163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.360122 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.360163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.360130 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq6qw\" (UniqueName: \"kubernetes.io/projected/05714c1e-3593-4b81-a683-fcba9d3438a7-kube-api-access-pq6qw\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:53.387517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.387479 4433 generic.go:356] "Generic (PLEG): container finished" podID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerID="806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351" exitCode=0 Oct 30 00:31:53.387710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.387531 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" Oct 30 00:31:53.387710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.387567 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" event={"ID":"05714c1e-3593-4b81-a683-fcba9d3438a7","Type":"ContainerDied","Data":"806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351"} Oct 30 00:31:53.387710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.387646 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m" event={"ID":"05714c1e-3593-4b81-a683-fcba9d3438a7","Type":"ContainerDied","Data":"18ac5322ee15e97126d76a1c95b8a410d06a9aed7dd738c52fe0e54f91037e7e"} Oct 30 00:31:53.387710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.387670 4433 scope.go:122] "RemoveContainer" containerID="806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351" Oct 30 00:31:53.399005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.398985 4433 scope.go:122] "RemoveContainer" containerID="806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351" Oct 30 00:31:53.399262 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:31:53.399243 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351\": container with ID starting with 806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351 not found: ID does not exist" containerID="806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351" Oct 30 00:31:53.399320 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.399270 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351"} err="failed to get container status \"806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351\": rpc error: code = NotFound desc = could not find container \"806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351\": container with ID starting with 806cfbbc0c089a9e32367c21a4a1dd838015b9278b5f7703b24e04bdb5663351 not found: ID does not exist" Oct 30 00:31:53.443832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.443786 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m"] Oct 30 00:31:53.449307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.449279 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-lch6m"] Oct 30 00:31:53.707968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.707878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:53.831664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:53.831372 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:54.666160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:54.666065 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:55.180762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:55.180725 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" path="/var/lib/kubelet/pods/05714c1e-3593-4b81-a683-fcba9d3438a7/volumes" Oct 30 00:31:56.139220 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:56.139186 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:56.977969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:56.977939 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:57.083022 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.082985 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:57.250856 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.250778 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk"] Oct 30 00:31:57.251289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.250982 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" podUID="c4fa4b2c-7828-49b8-8a33-cec310f66d3d" containerName="maas-api" containerID="cri-o://4ff1e4ba32c523d0b8e8f4b7b98728263201c3dcae193c95cd19cea7d0cab6ee" gracePeriod=30 Oct 30 00:31:57.312212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.312175 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:57.421464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.421425 4433 generic.go:356] "Generic (PLEG): container finished" podID="c4fa4b2c-7828-49b8-8a33-cec310f66d3d" containerID="4ff1e4ba32c523d0b8e8f4b7b98728263201c3dcae193c95cd19cea7d0cab6ee" exitCode=0 Oct 30 00:31:57.421695 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.421589 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" event={"ID":"c4fa4b2c-7828-49b8-8a33-cec310f66d3d","Type":"ContainerDied","Data":"4ff1e4ba32c523d0b8e8f4b7b98728263201c3dcae193c95cd19cea7d0cab6ee"} Oct 30 00:31:57.461048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.461011 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:57.565923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.565900 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:57.589705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.589676 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:31:57.631593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.630790 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-kube-api-access-gpdc9\" (UniqueName: \"kubernetes.io/projected/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-kube-api-access-gpdc9\") pod \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " Oct 30 00:31:57.631593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.630937 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-openshift-service-ca\") pod \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " Oct 30 00:31:57.631593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.631108 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-maas-api-tls\") pod \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " Oct 30 00:31:57.631593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.631226 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-metrics-certs\") pod \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\" (UID: \"c4fa4b2c-7828-49b8-8a33-cec310f66d3d\") " Oct 30 00:31:57.636601 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.632768 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-openshift-service-ca" pod "c4fa4b2c-7828-49b8-8a33-cec310f66d3d" (UID: "c4fa4b2c-7828-49b8-8a33-cec310f66d3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:57.663603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.662843 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-kube-api-access-gpdc9" pod "c4fa4b2c-7828-49b8-8a33-cec310f66d3d" (UID: "c4fa4b2c-7828-49b8-8a33-cec310f66d3d"). InnerVolumeSpecName "kube-api-access-gpdc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:57.678605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.676019 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-metrics-certs" pod "c4fa4b2c-7828-49b8-8a33-cec310f66d3d" (UID: "c4fa4b2c-7828-49b8-8a33-cec310f66d3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:57.702623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.702360 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-maas-api-tls" pod "c4fa4b2c-7828-49b8-8a33-cec310f66d3d" (UID: "c4fa4b2c-7828-49b8-8a33-cec310f66d3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:57.758607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.756219 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:57.758607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.756255 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:57.758607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.756270 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:57.758607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.756285 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdc9\" (UniqueName: \"kubernetes.io/projected/c4fa4b2c-7828-49b8-8a33-cec310f66d3d-kube-api-access-gpdc9\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:31:57.785633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.782809 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:57.908954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:57.908920 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:58.021072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.020354 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:58.290117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.290075 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:58.398171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.398123 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:58.430019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.429989 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" Oct 30 00:31:58.430209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.430016 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk" event={"ID":"c4fa4b2c-7828-49b8-8a33-cec310f66d3d","Type":"ContainerDied","Data":"0e0e2c04d9cefd11d1412d6ed4f3af17905a5aa972fb9104a6ec7c04f1008868"} Oct 30 00:31:58.430209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.430082 4433 scope.go:122] "RemoveContainer" containerID="4ff1e4ba32c523d0b8e8f4b7b98728263201c3dcae193c95cd19cea7d0cab6ee" Oct 30 00:31:58.486038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.486007 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk"] Oct 30 00:31:58.499150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.499123 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f649b0-6bc97cdddf-bwhjk"] Oct 30 00:31:58.507809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.507774 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:58.613806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.613711 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96"] Oct 30 00:31:58.614009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.613986 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" containerID="cri-o://587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70" gracePeriod=30 Oct 30 00:31:58.641724 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:58.641694 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:59.180804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:59.180765 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fa4b2c-7828-49b8-8a33-cec310f66d3d" path="/var/lib/kubelet/pods/c4fa4b2c-7828-49b8-8a33-cec310f66d3d/volumes" Oct 30 00:31:59.605827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:31:59.605794 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:01.144166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:01.144132 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:02.396696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:02.396657 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:02.859800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:02.859762 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:02.966188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:02.966148 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:03.078369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:03.078331 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:03.470116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:03.470083 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:03.900021 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:03.899993 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:32:04.011464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.011370 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-certs\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.011464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.011450 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0152a495-2048-4d28-8aa4-2bde079180e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0152a495-2048-4d28-8aa4-2bde079180e2-istio-podinfo\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.011762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.011548 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-kube-api-access-l78m4\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-kube-api-access-l78m4\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.012482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.012455 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-istio-token\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-istio-token\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.013234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.013212 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-envoy\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.013437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.013405 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-data\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.013563 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.013490 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0152a495-2048-4d28-8aa4-2bde079180e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0152a495-2048-4d28-8aa4-2bde079180e2-istiod-ca-cert\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.013563 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.013532 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-socket\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.013726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.013636 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-credential-socket\") pod \"0152a495-2048-4d28-8aa4-2bde079180e2\" (UID: \"0152a495-2048-4d28-8aa4-2bde079180e2\") " Oct 30 00:32:04.015289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.012848 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-certs" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.015289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.015008 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0152a495-2048-4d28-8aa4-2bde079180e2-istiod-ca-cert" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:04.015289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.015255 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-socket" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.015697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.015677 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-data" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.015933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.015912 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-credential-socket" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.019376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.019324 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-istio-token" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:04.019962 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.019941 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0152a495-2048-4d28-8aa4-2bde079180e2-istio-podinfo" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:04.021297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.021261 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-kube-api-access-l78m4" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "kube-api-access-l78m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:04.024831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.024805 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-envoy" pod "0152a495-2048-4d28-8aa4-2bde079180e2" (UID: "0152a495-2048-4d28-8aa4-2bde079180e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.115598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115553 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115595 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0152a495-2048-4d28-8aa4-2bde079180e2-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115604 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78m4\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-kube-api-access-l78m4\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115613 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0152a495-2048-4d28-8aa4-2bde079180e2-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115621 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115628 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115635 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0152a495-2048-4d28-8aa4-2bde079180e2-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115643 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.115830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.115650 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0152a495-2048-4d28-8aa4-2bde079180e2-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.474464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.474420 4433 generic.go:356] "Generic (PLEG): container finished" podID="0152a495-2048-4d28-8aa4-2bde079180e2" containerID="587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70" exitCode=0 Oct 30 00:32:04.474464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.474471 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" Oct 30 00:32:04.475072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.474507 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" event={"ID":"0152a495-2048-4d28-8aa4-2bde079180e2","Type":"ContainerDied","Data":"587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70"} Oct 30 00:32:04.475072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.474591 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96" event={"ID":"0152a495-2048-4d28-8aa4-2bde079180e2","Type":"ContainerDied","Data":"b9aa57edf3a5afe64089cd3c3c45cf498bb62ab106d2b150e92af6a18b84a838"} Oct 30 00:32:04.475072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.474623 4433 scope.go:122] "RemoveContainer" containerID="587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70" Oct 30 00:32:04.490363 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.490333 4433 scope.go:122] "RemoveContainer" containerID="587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70" Oct 30 00:32:04.491042 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:32:04.491014 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70\": container with ID starting with 587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70 not found: ID does not exist" containerID="587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70" Oct 30 00:32:04.491129 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.491053 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70"} err="failed to get container status \"587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70\": rpc error: code = NotFound desc = could not find container \"587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70\": container with ID starting with 587cfebc461e4e2c38ce4851c2346dbea7df58897dcb3604a67c08dc43b33e70 not found: ID does not exist" Oct 30 00:32:04.497388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.497364 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:04.530755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.530723 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96"] Oct 30 00:32:04.569637 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:04.567620 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-krx96"] Oct 30 00:32:05.016053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:05.016021 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:05.132132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:05.132096 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:05.180259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:05.180227 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" path="/var/lib/kubelet/pods/0152a495-2048-4d28-8aa4-2bde079180e2/volumes" Oct 30 00:32:05.326168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:05.326129 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz"] Oct 30 00:32:05.326458 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:05.326428 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" containerID="cri-o://3de5cd8b57485279691d47a36940db6ad1e2cb12400cbf99b503766a735adf10" gracePeriod=30 Oct 30 00:32:05.362867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:05.362823 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:06.060298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:06.060260 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:07.102302 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:32:07.102251 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:32:07.485190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:07.485098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.784076 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:08.784035 4433 patch_prober.go:29] interesting pod/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:32:08.784535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:08.784105 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:32:08.969424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:08.969390 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:09.696352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:09.696308 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb"] Oct 30 00:32:09.803626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:09.803557 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:09.932066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:09.932028 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:10.231354 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.231314 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:10.452436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.452366 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:10.625606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.625550 4433 generic.go:356] "Generic (PLEG): container finished" podID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerID="3de5cd8b57485279691d47a36940db6ad1e2cb12400cbf99b503766a735adf10" exitCode=0 Oct 30 00:32:10.626063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.626035 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" event={"ID":"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3","Type":"ContainerDied","Data":"3de5cd8b57485279691d47a36940db6ad1e2cb12400cbf99b503766a735adf10"} Oct 30 00:32:10.626292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.626276 4433 scope.go:122] "RemoveContainer" containerID="a78292dcc1fa741ea73e20651f88982aee80a5b446108df76aac0c5ea77ab098" Oct 30 00:32:10.706987 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.706958 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:32:10.823396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.823295 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-kube-api-access-gzmxk\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-kube-api-access-gzmxk\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.823437 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istiod-ca-cert\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.823483 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-credential-socket\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.823722 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-podinfo\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.823809 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-certs\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.823869 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-socket\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.823949 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-data\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.824033 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-envoy\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.824744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.824451 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-token\") pod \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\" (UID: \"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3\") " Oct 30 00:32:10.825164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.825133 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istiod-ca-cert" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:10.825398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.825371 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.825708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.825670 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-credential-socket" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:10.825886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.825866 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-data" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:10.826075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.826053 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-certs" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:10.826256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.826236 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-socket" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:10.830089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.828423 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-kube-api-access-gzmxk" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "kube-api-access-gzmxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:10.831153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.831087 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-envoy" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:10.831153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.831102 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-token" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:10.848325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.848281 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-podinfo" pod "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" (UID: "9d5c34ff-6315-43bd-af4e-67ca3bb5caa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:10.927236 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927195 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.927236 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927236 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.927470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927249 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.927470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927263 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.927470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927277 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.927470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927289 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.927470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927301 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzmxk\" (UniqueName: \"kubernetes.io/projected/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-kube-api-access-gzmxk\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:10.927470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:10.927315 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:11.635119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:11.635089 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" Oct 30 00:32:11.635119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:11.635109 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz" event={"ID":"9d5c34ff-6315-43bd-af4e-67ca3bb5caa3","Type":"ContainerDied","Data":"c74853cc46a3c7bfe6f40b397328f8a0d88a8c322ceb97ad79823dd6d8adb08d"} Oct 30 00:32:11.635360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:11.635166 4433 scope.go:122] "RemoveContainer" containerID="3de5cd8b57485279691d47a36940db6ad1e2cb12400cbf99b503766a735adf10" Oct 30 00:32:11.747783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:11.747746 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz"] Oct 30 00:32:11.777284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:11.777236 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f649b0-openshift-default-64bd455f9f-66csz"] Oct 30 00:32:12.057379 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:12.057339 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:13.182662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:13.182626 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" path="/var/lib/kubelet/pods/9d5c34ff-6315-43bd-af4e-67ca3bb5caa3/volumes" Oct 30 00:32:13.414016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:13.413983 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:14.697656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:14.697597 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" podUID="2f747d08-31d5-4400-be08-f8558e5f15ea" containerName="payload-pre-processing" containerID="cri-o://e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df" gracePeriod=25 Oct 30 00:32:14.765196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:14.765161 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:15.032899 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.032872 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:32:15.093321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.093286 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f747d08-31d5-4400-be08-f8558e5f15ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f747d08-31d5-4400-be08-f8558e5f15ea-plugins-config-volume\") pod \"2f747d08-31d5-4400-be08-f8558e5f15ea\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " Oct 30 00:32:15.093532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.093356 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f747d08-31d5-4400-be08-f8558e5f15ea-kube-api-access-sqwwv\" (UniqueName: \"kubernetes.io/projected/2f747d08-31d5-4400-be08-f8558e5f15ea-kube-api-access-sqwwv\") pod \"2f747d08-31d5-4400-be08-f8558e5f15ea\" (UID: \"2f747d08-31d5-4400-be08-f8558e5f15ea\") " Oct 30 00:32:15.093818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.093780 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f747d08-31d5-4400-be08-f8558e5f15ea-plugins-config-volume" pod "2f747d08-31d5-4400-be08-f8558e5f15ea" (UID: "2f747d08-31d5-4400-be08-f8558e5f15ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:15.094213 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.094188 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f747d08-31d5-4400-be08-f8558e5f15ea-plugins-config-volume\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:15.096265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.096237 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f747d08-31d5-4400-be08-f8558e5f15ea-kube-api-access-sqwwv" pod "2f747d08-31d5-4400-be08-f8558e5f15ea" (UID: "2f747d08-31d5-4400-be08-f8558e5f15ea"). InnerVolumeSpecName "kube-api-access-sqwwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:15.195482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.195451 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqwwv\" (UniqueName: \"kubernetes.io/projected/2f747d08-31d5-4400-be08-f8558e5f15ea-kube-api-access-sqwwv\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:15.255135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.255040 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:15.666334 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.666301 4433 generic.go:356] "Generic (PLEG): container finished" podID="2f747d08-31d5-4400-be08-f8558e5f15ea" containerID="e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df" exitCode=0 Oct 30 00:32:15.666515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.666346 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" Oct 30 00:32:15.666515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.666381 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" event={"ID":"2f747d08-31d5-4400-be08-f8558e5f15ea","Type":"ContainerDied","Data":"e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df"} Oct 30 00:32:15.666515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.666436 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb" event={"ID":"2f747d08-31d5-4400-be08-f8558e5f15ea","Type":"ContainerDied","Data":"c2dd0232ff631735c6aa1c79fcc77ae961f1c94832d357d3b897d5a374416a85"} Oct 30 00:32:15.666515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.666457 4433 scope.go:122] "RemoveContainer" containerID="e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df" Oct 30 00:32:15.676933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.676909 4433 scope.go:122] "RemoveContainer" containerID="e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df" Oct 30 00:32:15.677205 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:32:15.677185 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df\": container with ID starting with e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df not found: ID does not exist" containerID="e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df" Oct 30 00:32:15.677271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.677213 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df"} err="failed to get container status \"e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df\": rpc error: code = NotFound desc = could not find container \"e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df\": container with ID starting with e5eea2265947a5b6be1db6c4239324d684e913a80f9a1d424b784b350dd2e6df not found: ID does not exist" Oct 30 00:32:15.726066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.726017 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb"] Oct 30 00:32:15.740259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.740224 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3e13d-754b458bcf-hj4nb"] Oct 30 00:32:15.976763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.976680 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt"] Oct 30 00:32:15.976944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:15.976910 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" podUID="c5ac6944-cb98-4e2f-a35e-4662ecf59e41" containerName="maas-api" containerID="cri-o://d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130" gracePeriod=30 Oct 30 00:32:16.039135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.039098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:16.291070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.290507 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:32:16.418501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.418446 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-metrics-certs\") pod \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " Oct 30 00:32:16.419663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.419637 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-kube-api-access-lx9sg\" (UniqueName: \"kubernetes.io/projected/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-kube-api-access-lx9sg\") pod \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " Oct 30 00:32:16.419823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.419713 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-openshift-service-ca\") pod \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " Oct 30 00:32:16.419823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.419791 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-maas-api-tls\") pod \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\" (UID: \"c5ac6944-cb98-4e2f-a35e-4662ecf59e41\") " Oct 30 00:32:16.420356 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.420330 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-openshift-service-ca" pod "c5ac6944-cb98-4e2f-a35e-4662ecf59e41" (UID: "c5ac6944-cb98-4e2f-a35e-4662ecf59e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:16.420502 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.420478 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:16.421027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.421003 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-metrics-certs" pod "c5ac6944-cb98-4e2f-a35e-4662ecf59e41" (UID: "c5ac6944-cb98-4e2f-a35e-4662ecf59e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:16.422359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.422233 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-maas-api-tls" pod "c5ac6944-cb98-4e2f-a35e-4662ecf59e41" (UID: "c5ac6944-cb98-4e2f-a35e-4662ecf59e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:16.423108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.422446 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-kube-api-access-lx9sg" pod "c5ac6944-cb98-4e2f-a35e-4662ecf59e41" (UID: "c5ac6944-cb98-4e2f-a35e-4662ecf59e41"). InnerVolumeSpecName "kube-api-access-lx9sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:16.483226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.483188 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:16.521545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.521464 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9sg\" (UniqueName: \"kubernetes.io/projected/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-kube-api-access-lx9sg\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:16.521545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.521497 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:16.521545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.521508 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6944-cb98-4e2f-a35e-4662ecf59e41-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:32:16.602641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.596764 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:16.675149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.675112 4433 generic.go:356] "Generic (PLEG): container finished" podID="c5ac6944-cb98-4e2f-a35e-4662ecf59e41" containerID="d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130" exitCode=0 Oct 30 00:32:16.675350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.675163 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" Oct 30 00:32:16.675350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.675196 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" event={"ID":"c5ac6944-cb98-4e2f-a35e-4662ecf59e41","Type":"ContainerDied","Data":"d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130"} Oct 30 00:32:16.675350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.675257 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt" event={"ID":"c5ac6944-cb98-4e2f-a35e-4662ecf59e41","Type":"ContainerDied","Data":"9fa1fe5920d9e74493f4fc37cc9059f3968addada1c5d5395fcce5396b6842a0"} Oct 30 00:32:16.675350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.675281 4433 scope.go:122] "RemoveContainer" containerID="d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130" Oct 30 00:32:16.686524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.686502 4433 scope.go:122] "RemoveContainer" containerID="d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130" Oct 30 00:32:16.686869 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:32:16.686845 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130\": container with ID starting with d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130 not found: ID does not exist" containerID="d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130" Oct 30 00:32:16.686973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.686873 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130"} err="failed to get container status \"d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130\": rpc error: code = NotFound desc = could not find container \"d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130\": container with ID starting with d93ea2344b90b146a9ee87353c2659b0a8b365302a5e313595937c22d35b9130 not found: ID does not exist" Oct 30 00:32:16.703531 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.703502 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:16.740713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.740561 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt"] Oct 30 00:32:16.749266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:16.749235 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3e13d-7ffbffdb4d-rvldt"] Oct 30 00:32:17.181058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:17.181021 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f747d08-31d5-4400-be08-f8558e5f15ea" path="/var/lib/kubelet/pods/2f747d08-31d5-4400-be08-f8558e5f15ea/volumes" Oct 30 00:32:17.181634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:17.181618 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ac6944-cb98-4e2f-a35e-4662ecf59e41" path="/var/lib/kubelet/pods/c5ac6944-cb98-4e2f-a35e-4662ecf59e41/volumes" Oct 30 00:32:17.282278 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:17.282239 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:18.375529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:18.375480 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:19.833406 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:19.833367 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:21.210164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:21.210125 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:21.373508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:21.373474 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:21.485316 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:21.485046 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:22.858636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:22.858522 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:23.417597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:23.417406 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:23.534598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:23.531203 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:23.744330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:23.742719 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:24.399551 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:24.399519 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:24.930933 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:24.930885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:25.035131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:25.035093 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:25.234817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:25.234719 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:25.587352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:25.587317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:26.693398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:26.693357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:28.048628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:28.048561 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:29.410142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:29.410095 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:30.842445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:30.842408 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:31.757123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:31.757082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:31.860392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:31.860357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:32.106417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:32.106368 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:32.263597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:32.254458 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:33.673359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:33.673323 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:35.262670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:35.262585 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:36.754057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:36.754026 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:37.106151 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:32:37.106101 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:32:38.181440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:38.181402 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:39.083561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:39.083526 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:39.343234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:39.343138 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:39.565182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:39.565140 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:41.073168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:41.073129 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:42.462283 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:42.462250 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:42.582116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:42.582077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:43.068921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:43.068885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:43.445206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:43.445118 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:43.971309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:43.971266 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:45.286439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:45.286401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:45.571111 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:45.571061 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:45.896436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:45.896322 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:46.222691 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:46.222566 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:47.022648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:47.022609 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:47.129539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:47.129504 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:47.271983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:47.271942 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:47.635122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:47.635087 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:48.196710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:48.196669 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:48.320139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:48.320101 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:48.732179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:48.732139 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:48.866680 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:48.866639 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:50.143844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:50.143797 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:50.255272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:50.255237 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:50.645243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:50.645203 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:51.678726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:51.678685 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:52.051023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.050871 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:52.771610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.770012 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:52.808600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.806673 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:52.822470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.809893 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:52.891603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.889893 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:32:52.925144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.906235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:52.925144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.906327 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:52.990712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:52.990661 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:53.081599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.077771 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:32:53.096609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.093436 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:53.182335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.182291 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:53.185566 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.182605 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:53.517655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.511988 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:32:53.517655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.512049 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:53.649608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.647083 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:32:53.820654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.786276 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:53.895752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.895693 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:53.928600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.896950 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:32:53.941694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.912870 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:53.943073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:53.943049 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:32:54.125420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.125321 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.141533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.125696 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.143063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.143037 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:32:54.200297 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.200251 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.200972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.200948 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:32:54.280607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.280557 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:32:54.319024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.304566 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:32:54.319024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.318965 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:32:54.324774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.324735 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:32:54.639779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.639742 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:54.760500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:54.760407 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:55.660393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:55.660354 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:57.241855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:57.241807 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:58.637835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:58.637801 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:59.209734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:59.209700 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:59.337999 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:32:59.337959 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:00.330588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:00.330531 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:01.544708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:01.544667 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:01.653524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:01.653487 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:02.233649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:02.233610 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:02.337921 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:02.337876 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:03.670132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:03.670089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:05.122551 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:05.122516 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:05.237941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:05.237868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:05.342030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:05.341998 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:06.657969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:06.657933 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:07.101138 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:33:07.101103 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:33:08.164938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:08.164891 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:09.597664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:09.597619 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:09.715045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:09.715006 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:09.824428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:09.824390 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:11.053242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:11.053204 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:12.491040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:12.491006 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:12.865325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:12.865287 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:13.371395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:13.371356 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:13.494326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:13.494242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:14.086497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:14.086462 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:15.549817 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:15.549776 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:15.651273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:15.651240 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:16.610336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.610301 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4"] Oct 30 00:33:16.611233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611208 4433 cpu_manager.go:414] "containerMap: removing container" podUID="c4fa4b2c-7828-49b8-8a33-cec310f66d3d" containerName="maas-api" Oct 30 00:33:16.611233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611232 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fa4b2c-7828-49b8-8a33-cec310f66d3d" containerName="maas-api" Oct 30 00:33:16.611233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611252 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" Oct 30 00:33:16.611233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611259 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611279 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2f747d08-31d5-4400-be08-f8558e5f15ea" containerName="payload-pre-processing" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611288 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f747d08-31d5-4400-be08-f8558e5f15ea" containerName="payload-pre-processing" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611304 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611311 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611322 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611330 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611363 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611370 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611407 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611413 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611426 4433 cpu_manager.go:414] "containerMap: removing container" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611434 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611453 4433 cpu_manager.go:414] "containerMap: removing container" podUID="c5ac6944-cb98-4e2f-a35e-4662ecf59e41" containerName="maas-api" Oct 30 00:33:16.611622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611460 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ac6944-cb98-4e2f-a35e-4662ecf59e41" containerName="maas-api" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611660 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611674 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="05714c1e-3593-4b81-a683-fcba9d3438a7" containerName="istio-proxy" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611688 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ac6944-cb98-4e2f-a35e-4662ecf59e41" containerName="maas-api" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611707 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fa4b2c-7828-49b8-8a33-cec310f66d3d" containerName="maas-api" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611715 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0152a495-2048-4d28-8aa4-2bde079180e2" containerName="istio-proxy" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611726 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f747d08-31d5-4400-be08-f8558e5f15ea" containerName="payload-pre-processing" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611735 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" Oct 30 00:33:16.612329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.611744 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebc67f8-bd38-44fd-ab6d-42b50bd609fc" containerName="istio-proxy" Oct 30 00:33:16.615745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.615724 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.645618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.641781 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4"] Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.737235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.741695 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cmw\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-kube-api-access-t5cmw\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.741760 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.741806 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.741890 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.742000 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.742054 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.742113 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.742209 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.744595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.742268 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.840851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.840815 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:16.843341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843312 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843486 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843368 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cmw\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-kube-api-access-t5cmw\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843486 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843417 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843480 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843599 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843731 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843798 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843804 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843859 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843916 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.843963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.843921 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.844123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.844033 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.844123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.844101 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.844431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.844408 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.846310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.846282 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.846585 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.846554 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.867636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.867533 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.867761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.867682 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cmw\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-kube-api-access-t5cmw\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:16.932621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:16.932359 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:17.271090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.271063 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4"] Oct 30 00:33:17.271306 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:33:17.271285 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f69cc2_29c2_47f4_8b74_34ace130d3d8.slice/crio-0f20eb5620b07378ac4c0ad59bce3f3f4f9bc74368a769bf5b18525b5b6d81fd WatchSource:0}: Error finding container 0f20eb5620b07378ac4c0ad59bce3f3f4f9bc74368a769bf5b18525b5b6d81fd: Status 404 returned error can't find the container with id 0f20eb5620b07378ac4c0ad59bce3f3f4f9bc74368a769bf5b18525b5b6d81fd Oct 30 00:33:17.273889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.273853 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:17.274013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.273929 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:17.274013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.273975 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:17.379991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.379947 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:17.680841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.680798 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" event={"ID":"a3f69cc2-29c2-47f4-8b74-34ace130d3d8","Type":"ContainerStarted","Data":"6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7"} Oct 30 00:33:17.681330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.680866 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" event={"ID":"a3f69cc2-29c2-47f4-8b74-34ace130d3d8","Type":"ContainerStarted","Data":"0f20eb5620b07378ac4c0ad59bce3f3f4f9bc74368a769bf5b18525b5b6d81fd"} Oct 30 00:33:17.789561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.789504 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podStartSLOduration=1.7894902240000001 podStartE2EDuration="1.789490224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:17.772818123 +0000 UTC m=+1706.706886672" watchObservedRunningTime="2026-10-30 00:33:17.789490224 +0000 UTC m=+1706.723558758" Oct 30 00:33:17.933082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:17.932966 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:18.934106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:18.934074 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:33:18.934491 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:18.934128 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:33:19.179648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:19.179608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:19.388351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:19.388310 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:19.492639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:19.492598 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:19.932890 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:19.932859 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:33:19.933080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:19.932922 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:33:20.933326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:20.933287 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:33:20.933886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:20.933362 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:33:21.017095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:21.017056 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:21.933361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:21.933320 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:33:21.933816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:21.933389 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:33:22.553737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:22.553696 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:22.933246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:22.933215 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:33:22.933521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:22.933496 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:33:23.933549 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:23.933508 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:33:23.934238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:23.934197 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:33:24.697685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:24.691987 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:25.976428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:25.976391 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:25.977127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:25.977083 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:26.252656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.222793 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:33:26.252656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.223589 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:26.252656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.223913 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:33:26.252656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.236734 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:33:26.468251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.436329 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:26.468251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.451238 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:26.468251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.466446 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:33:26.468251 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.467112 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:33:26.541501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.528659 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:33:26.801655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:26.791706 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.014504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.010026 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:27.014504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.010090 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.280598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.261000 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:33:27.290692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.290646 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:33:27.306041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.292648 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:33:27.344597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.306878 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:33:27.453774 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.453724 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:33:27.571657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.560491 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:33:27.571657 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.560555 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:33:27.579598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.577427 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:33:27.601242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.600655 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:33:27.601242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.600969 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:33:27.606797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.606662 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.606797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.606746 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:33:27.665745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.646427 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780392 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780457 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780493 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780528 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780565 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780641 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780675 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:33:27.782632 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.780870 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:27.804258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.804215 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:33:27.804612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.804560 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.825689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:27.819892 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:33:28.102658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.099033 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 00:33:28.102658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.099483 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:28.102658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.099525 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:28.344599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.340947 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:33:28.477039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.476932 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:28.762693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.548448 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.042655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.736644 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.093599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.740719 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:33:29.272742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.954353 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.273522 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.954393 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.273810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.954676 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.277667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.954883 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.277667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.955011 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.277667 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:28.955146 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.387761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.336684 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:29.387761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.336750 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.387761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.350688 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.387761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.351473 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:33:29.409878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.405183 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:29.777635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.777505 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.791702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.791648 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:33:29.905984 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:29.899950 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.019894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.019855 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:30.070700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.070654 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.092862 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.092828 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:33:30.093149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.093130 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:33:30.093326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.093311 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:33:30.093495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.093481 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:30.097127 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.097105 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:30.098298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.098280 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:33:30.189314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.189273 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:33:30.198096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.190303 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:30.215344 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.215307 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:33:30.293598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.277739 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.293598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.278526 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.293598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.279161 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.293598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.280393 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:33:30.293598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.281016 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.367167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.356724 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 00:33:30.383698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.383663 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:30.428597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.426362 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:33:30.436744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.436686 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.455348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.454657 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:33:30.468497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.468459 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:33:30.475562 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.475526 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:33:30.568327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.566378 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:33:30.597147 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.597106 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:33:30.608871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.608842 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:33:30.613018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.612994 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:33:30.639033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.638952 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:33:30.684382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.684349 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:33:30.732624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:30.732559 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:31.198280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.198243 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:31.643763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.643717 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v"] Oct 30 00:33:31.644703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.644673 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5c34ff-6315-43bd-af4e-67ca3bb5caa3" containerName="istio-proxy" Oct 30 00:33:31.648973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.648949 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.667065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.666473 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l"] Oct 30 00:33:31.677294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.677223 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.681196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.681168 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v"] Oct 30 00:33:31.701771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.701740 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l"] Oct 30 00:33:31.715306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.715275 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:31.751220 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751192 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751338 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751267 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751372 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751521 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57chk\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-kube-api-access-57chk\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751599 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751693 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751731 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751802 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.751880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.751863 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.853959 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.854232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854027 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.854232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854063 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854192 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57chk\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-kube-api-access-57chk\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854223 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.854469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854278 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854309 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.854469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854362 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854398 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.854469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854434 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854486 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.854830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854527 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.854830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854592 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854643 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854747 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.854830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854788 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.855106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854881 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.855106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.854937 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nd7\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-kube-api-access-f5nd7\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.857095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.856963 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.857561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.857536 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.858237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.858212 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.858463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.858443 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.859398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.859370 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.862013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.861975 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.862141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.862118 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.888054 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.888017 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57chk\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-kube-api-access-57chk\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.888393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.888369 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.903096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.903027 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:31.956564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.956531 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.956776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.956646 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nd7\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-kube-api-access-f5nd7\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.956776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.956704 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.956776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.956740 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.956948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.956921 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.957071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.957035 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.957196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.957159 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.957539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.957272 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.957745 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.957361 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.957874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.957794 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.957931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.957791 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.957931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.957709 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.958033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.958016 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.958358 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.958325 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.959250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.959222 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.960100 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.960079 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.968037 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.968014 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:31.982347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.982320 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:31.982450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:31.982366 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nd7\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-kube-api-access-f5nd7\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:32.011147 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.011107 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:32.013252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.013226 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:32.298822 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:33:32.298786 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75459eb3_1ef0_4e11_88cc_bd9ce9c3ac2d.slice/crio-7162bbc38810bfbd0e2a2ac5d5d6f8e31c74b6efebcd94350b8bd6ffe528128f WatchSource:0}: Error finding container 7162bbc38810bfbd0e2a2ac5d5d6f8e31c74b6efebcd94350b8bd6ffe528128f: Status 404 returned error can't find the container with id 7162bbc38810bfbd0e2a2ac5d5d6f8e31c74b6efebcd94350b8bd6ffe528128f Oct 30 00:33:32.300692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.300669 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v"] Oct 30 00:33:32.302410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.302377 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:32.302493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.302445 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:32.302493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.302473 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:32.383033 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:33:32.382995 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d12bf49_df66_40a5_b4a6_74163b6a88ed.slice/crio-9a8a8e27bb971da433d55365d7c52f726b9279de8de7c35dc9ccd1f9d20486cb WatchSource:0}: Error finding container 9a8a8e27bb971da433d55365d7c52f726b9279de8de7c35dc9ccd1f9d20486cb: Status 404 returned error can't find the container with id 9a8a8e27bb971da433d55365d7c52f726b9279de8de7c35dc9ccd1f9d20486cb Oct 30 00:33:32.383160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.383028 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l"] Oct 30 00:33:32.386128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.386090 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:32.386206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.386179 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:32.386241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.386228 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:32.860173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.860136 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr"] Oct 30 00:33:32.865523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.865500 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.902155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.902123 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr"] Oct 30 00:33:32.994105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994018 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994078 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994130 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrmz\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-kube-api-access-hjrmz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994384 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994158 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994539 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994730 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994760 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994781 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:32.994815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:32.994812 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.019356 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.019310 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" event={"ID":"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d","Type":"ContainerStarted","Data":"7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec"} Oct 30 00:33:33.019634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.019379 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" event={"ID":"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d","Type":"ContainerStarted","Data":"7162bbc38810bfbd0e2a2ac5d5d6f8e31c74b6efebcd94350b8bd6ffe528128f"} Oct 30 00:33:33.021242 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.021216 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" event={"ID":"7d12bf49-df66-40a5-b4a6-74163b6a88ed","Type":"ContainerStarted","Data":"966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7"} Oct 30 00:33:33.021361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.021258 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" event={"ID":"7d12bf49-df66-40a5-b4a6-74163b6a88ed","Type":"ContainerStarted","Data":"9a8a8e27bb971da433d55365d7c52f726b9279de8de7c35dc9ccd1f9d20486cb"} Oct 30 00:33:33.069531 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.067271 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podStartSLOduration=2.067252045 podStartE2EDuration="2.067252045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:33.064186171 +0000 UTC m=+1721.998254736" watchObservedRunningTime="2026-10-30 00:33:33.067252045 +0000 UTC m=+1722.001320581" Oct 30 00:33:33.096939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.096906 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.096963 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097050 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrmz\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-kube-api-access-hjrmz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097089 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097233 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097458 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097493 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097541 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097721 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097589 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097721 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097653 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097832 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.097926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097903 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.098008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.097977 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.098124 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.098094 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.099380 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.099359 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.100210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.100193 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.120805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.120741 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.120980 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.120950 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrmz\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-kube-api-access-hjrmz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.126802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.126744 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podStartSLOduration=2.126726411 podStartE2EDuration="2.126726411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:33.121456911 +0000 UTC m=+1722.055525510" watchObservedRunningTime="2026-10-30 00:33:33.126726411 +0000 UTC m=+1722.060794947" Oct 30 00:33:33.282430 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.282374 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:33.634678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.634644 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr"] Oct 30 00:33:33.636434 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:33:33.636406 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fb91fd_aa1a_43a5_a388_24ba8562fa23.slice/crio-cdd0aa6b8011679c5679eaacbf4281076b852517a3f417ef41915e27636df98f WatchSource:0}: Error finding container cdd0aa6b8011679c5679eaacbf4281076b852517a3f417ef41915e27636df98f: Status 404 returned error can't find the container with id cdd0aa6b8011679c5679eaacbf4281076b852517a3f417ef41915e27636df98f Oct 30 00:33:33.640715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.640675 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:33.640838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.640762 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:33.640838 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.640804 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:33.844730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.844694 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:33.976337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.969954 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:33.986595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.978691 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:33.986595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.978820 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:33.986595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:33.979616 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:34.022829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.017863 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:34.029770 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.029732 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:34.029968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.029795 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:34.120134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.115162 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" event={"ID":"d8fb91fd-aa1a-43a5-a388-24ba8562fa23","Type":"ContainerStarted","Data":"284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c"} Oct 30 00:33:34.120134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.115232 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" event={"ID":"d8fb91fd-aa1a-43a5-a388-24ba8562fa23","Type":"ContainerStarted","Data":"cdd0aa6b8011679c5679eaacbf4281076b852517a3f417ef41915e27636df98f"} Oct 30 00:33:34.131073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.130949 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:34.190798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.190737 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podStartSLOduration=2.190715035 podStartE2EDuration="2.190715035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:34.186040884 +0000 UTC m=+1723.120109460" watchObservedRunningTime="2026-10-30 00:33:34.190715035 +0000 UTC m=+1723.124783572" Oct 30 00:33:34.306656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.292293 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:34.308033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.307686 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:34.308033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.307763 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:34.987889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.975240 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:34.987889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:34.975324 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:35.045430 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.045392 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:35.045788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.045759 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:35.313470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.313433 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:35.313807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.313772 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:35.340595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.339901 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w"] Oct 30 00:33:35.427596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.427174 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:35.842595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.840225 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:35.993592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.987369 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:35.993592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.987704 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:35.993592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:35.987754 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:36.020381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.019023 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:36.020381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.019106 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:36.288655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.288295 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.294444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.291256 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.306047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.306009 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:36.306391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.306345 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:36.326343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.326305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:36.471290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.471244 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.610031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.550003 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:36.610031 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.550264 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.734598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.711431 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:36.991091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.990998 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:36.991901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:36.991864 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:37.022224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.022186 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:37.022886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.022850 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:37.070727 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.070682 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:37.250496 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:33:37.250389 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:33:37.292786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.291796 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:37.292786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.291855 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:37.307560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.307507 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:37.348971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.339536 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:37.376700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.349245 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:37.376700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.349628 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:33:37.578703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.563241 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:33:37.915325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.915232 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:37.973285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.973236 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:37.973555 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:37.973514 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:38.056596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:38.047955 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:38.056596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:38.048014 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:38.305692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:38.303216 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:38.305692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:38.303317 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:38.342213 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:38.341802 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:38.908690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:38.908654 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d"] Oct 30 00:33:39.010599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.000669 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:39.010599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.000729 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:39.044596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.011235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:39.048592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.021629 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.086630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.036162 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:39.111720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.111669 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:39.151604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.126392 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d"] Oct 30 00:33:39.162077 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.146475 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-credential-socket\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.250758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.250715 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-certs\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.258673 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-data\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.258812 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-envoy\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.258987 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-token\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.259047 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnpk\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-kube-api-access-jlnpk\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.259131 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istiod-ca-cert\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.259165 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-socket\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.259295 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-podinfo\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.197451 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:39.267218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.196407 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-05c6b2-openshift-default-dockercfg-4qxl6\"" Oct 30 00:33:39.496403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.496311 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:39.496682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.496649 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:39.608631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.579160 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:39.862325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.862281 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-token\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.897025 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.896736 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnpk\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-kube-api-access-jlnpk\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.920599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.897626 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istiod-ca-cert\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.989599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.980462 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-socket\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.989599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.920922 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.989599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.922398 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istiod-ca-cert\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.989599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.921490 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:39.989599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.981687 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-podinfo\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.989599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.982484 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-credential-socket\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:39.989599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.982851 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-certs\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.023154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:39.999459 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-data\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.088690 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.032760 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-socket\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.089307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.011615 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-certs\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.089517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.034903 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-data\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.089649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.011353 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-credential-socket\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.089760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.046793 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:40.089895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.089872 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:40.090007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.088265 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-envoy\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.126668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.102204 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-token\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.126668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.088121 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.126668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.046937 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:40.126668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.102360 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:40.126668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.111254 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnpk\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-kube-api-access-jlnpk\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.126668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.117297 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-envoy\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.221599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.219369 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-podinfo\") pod \"e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.285598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.285192 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:40.285598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.285255 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:40.286785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.286754 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.287291 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.286974 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.376598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.376024 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:40.515593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.515330 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:40.808339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.806671 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d"] Oct 30 00:33:40.811929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.811890 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:40.812046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.811978 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:40.812046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.812009 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:40.834169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.833743 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:40.901460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.899405 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" event={"ID":"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499","Type":"ContainerStarted","Data":"ed2c49768b4e14a8a055512da4c517580483316833281bff934168f22db6f61b"} Oct 30 00:33:40.972622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.971963 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 00:33:40.972622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:40.972019 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 00:33:41.013639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.013605 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:33:41.013775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.013664 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:33:41.283607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.282902 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:33:41.283607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.282973 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:33:41.909982 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.909939 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" event={"ID":"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499","Type":"ContainerStarted","Data":"7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51"} Oct 30 00:33:41.970978 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.970910 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podStartSLOduration=3.970895863 podStartE2EDuration="3.970895863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:41.968691058 +0000 UTC m=+1730.902759618" watchObservedRunningTime="2026-10-30 00:33:41.970895863 +0000 UTC m=+1730.904964505" Oct 30 00:33:41.975611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.975564 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:41.975982 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.975962 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:41.976747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:41.976725 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:33:42.171547 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.171457 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:42.172428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.172338 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:42.173567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.173547 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:33:42.379837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.379807 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:42.387639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.387566 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:42.439360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.439265 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:42.576694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.572482 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:42.682942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.682528 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:42.808435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.808401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:42.919440 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.919407 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:42.920398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.920377 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:33:42.949142 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.949111 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:42.949555 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.949533 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:42.950485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:42.950462 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:33:44.267557 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:44.267522 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:44.373683 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:44.373630 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:44.480341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:44.480305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:45.964985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:45.964943 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:46.133023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:46.132991 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:46.238012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:46.237929 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:47.870529 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:47.870490 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:48.336682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.336645 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn"] Oct 30 00:33:48.342027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.342005 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.364077 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.364046 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn"] Oct 30 00:33:48.372473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.372443 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a782e5-openshift-default-dockercfg-vtgkj\"" Oct 30 00:33:48.499812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.499769 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.499985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.499823 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-podinfo\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.500030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.499989 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-certs\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.500070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.500028 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643500a6-e1c6-4483-a4f6-ae816b5b8987-istiod-ca-cert\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.500185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.500164 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-data\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.500263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.500248 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-token\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.500322 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.500310 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-envoy\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.500360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.500341 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-credential-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.500482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.500469 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8th\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-kube-api-access-6z8th\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.601958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.601868 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-token\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.601958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.601945 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-envoy\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.601987 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-credential-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602113 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8th\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-kube-api-access-6z8th\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602168 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602198 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-podinfo\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602297 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-certs\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602328 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643500a6-e1c6-4483-a4f6-ae816b5b8987-istiod-ca-cert\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602416 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-data\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602492 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-credential-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602568 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602744 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-certs\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.602891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.602833 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-data\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.603301 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.603276 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643500a6-e1c6-4483-a4f6-ae816b5b8987-istiod-ca-cert\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.604991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.604968 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-envoy\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.605143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.605007 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-podinfo\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.628332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.628305 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-token\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.628627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.628599 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8th\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-kube-api-access-6z8th\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.658769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.658741 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:48.986900 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:33:48.986871 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643500a6_e1c6_4483_a4f6_ae816b5b8987.slice/crio-e07fe0c3ec8e3d3063a7cbab2c7a18dd8a30cb70b09da9b88beeb37854179aba WatchSource:0}: Error finding container e07fe0c3ec8e3d3063a7cbab2c7a18dd8a30cb70b09da9b88beeb37854179aba: Status 404 returned error can't find the container with id e07fe0c3ec8e3d3063a7cbab2c7a18dd8a30cb70b09da9b88beeb37854179aba Oct 30 00:33:48.987834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.987806 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn"] Oct 30 00:33:48.989729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.989693 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:48.989807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.989766 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:48.989844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:48.989807 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:33:49.502171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:49.502136 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:49.998630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:49.998588 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" event={"ID":"643500a6-e1c6-4483-a4f6-ae816b5b8987","Type":"ContainerStarted","Data":"3987de7ed327b26ec42aebf970bb47a2970b64633a1a0ba2620a16e4069b9ed1"} Oct 30 00:33:49.999080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:49.998662 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" event={"ID":"643500a6-e1c6-4483-a4f6-ae816b5b8987","Type":"ContainerStarted","Data":"e07fe0c3ec8e3d3063a7cbab2c7a18dd8a30cb70b09da9b88beeb37854179aba"} Oct 30 00:33:50.080593 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:50.079781 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" podStartSLOduration=2.079756629 podStartE2EDuration="2.079756629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:50.07266926 +0000 UTC m=+1739.006737796" watchObservedRunningTime="2026-10-30 00:33:50.079756629 +0000 UTC m=+1739.013825158" Oct 30 00:33:50.172596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:50.170132 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:50.620377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:50.620336 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:50.672597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:50.661795 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:50.672597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:50.661965 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:33:50.672597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:50.662024 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 00:33:51.104820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.104007 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:51.122118 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.122088 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84"] Oct 30 00:33:51.145438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.145400 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9"] Oct 30 00:33:51.218212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.218172 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:51.333650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.333616 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:51.447071 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.446985 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:51.696961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.696926 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:33:51.716175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:51.697172 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 00:33:52.277597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.264025 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" containerID="cri-o://c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c" gracePeriod=44 Oct 30 00:33:52.486654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.479338 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused" Oct 30 00:33:52.683597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.672300 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:33:52.683597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.672366 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 00:33:52.787679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.786373 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:33:52.836902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.836859 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tls-certs\") pod \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " Oct 30 00:33:52.837087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.836946 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-model-cache\") pod \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " Oct 30 00:33:52.838664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.838217 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-model-cache" pod "dc58932a-ae73-4b75-862b-30ebbdebfeb4" (UID: "dc58932a-ae73-4b75-862b-30ebbdebfeb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:52.840461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.840390 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tls-certs" pod "dc58932a-ae73-4b75-862b-30ebbdebfeb4" (UID: "dc58932a-ae73-4b75-862b-30ebbdebfeb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:52.939265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.939231 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc58932a-ae73-4b75-862b-30ebbdebfeb4-kube-api-access-l5v66\" (UniqueName: \"kubernetes.io/projected/dc58932a-ae73-4b75-862b-30ebbdebfeb4-kube-api-access-l5v66\") pod \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.939442 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-home\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-home\") pod \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.939607 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tmp-dir\") pod \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.939672 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-dshm\") pod \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\" (UID: \"dc58932a-ae73-4b75-862b-30ebbdebfeb4\") " Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.940564 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tmp-dir" pod "dc58932a-ae73-4b75-862b-30ebbdebfeb4" (UID: "dc58932a-ae73-4b75-862b-30ebbdebfeb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.940829 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-home" pod "dc58932a-ae73-4b75-862b-30ebbdebfeb4" (UID: "dc58932a-ae73-4b75-862b-30ebbdebfeb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.941558 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.941598 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.941611 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:33:52.942428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.941646 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:33:52.944422 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.944390 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-dshm" pod "dc58932a-ae73-4b75-862b-30ebbdebfeb4" (UID: "dc58932a-ae73-4b75-862b-30ebbdebfeb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:52.944861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:52.944835 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc58932a-ae73-4b75-862b-30ebbdebfeb4-kube-api-access-l5v66" pod "dc58932a-ae73-4b75-862b-30ebbdebfeb4" (UID: "dc58932a-ae73-4b75-862b-30ebbdebfeb4"). InnerVolumeSpecName "kube-api-access-l5v66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:53.025848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.025816 4433 generic.go:356] "Generic (PLEG): container finished" podID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerID="c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c" exitCode=0 Oct 30 00:33:53.026058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.025978 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" event={"ID":"dc58932a-ae73-4b75-862b-30ebbdebfeb4","Type":"ContainerDied","Data":"c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c"} Oct 30 00:33:53.026058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.026015 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" event={"ID":"dc58932a-ae73-4b75-862b-30ebbdebfeb4","Type":"ContainerDied","Data":"a7bbd524ac076446adaa5803c7046d2db66c0c24571847ebb4f8a150de93554d"} Oct 30 00:33:53.026058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.026035 4433 scope.go:122] "RemoveContainer" containerID="c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c" Oct 30 00:33:53.026233 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.026200 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w" Oct 30 00:33:53.038568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.038548 4433 scope.go:122] "RemoveContainer" containerID="c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c" Oct 30 00:33:53.038905 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:33:53.038881 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c\": container with ID starting with c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c not found: ID does not exist" containerID="c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c" Oct 30 00:33:53.039009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.038920 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c"} err="failed to get container status \"c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c\": rpc error: code = NotFound desc = could not find container \"c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c\": container with ID starting with c9231f7c832447389432fc6d49050dff96fa9b3092d861fafea2b559c10e304c not found: ID does not exist" Oct 30 00:33:53.042852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.042829 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc58932a-ae73-4b75-862b-30ebbdebfeb4-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:33:53.042954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.042857 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5v66\" (UniqueName: \"kubernetes.io/projected/dc58932a-ae73-4b75-862b-30ebbdebfeb4-kube-api-access-l5v66\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:33:53.092273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.092193 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w"] Oct 30 00:33:53.097190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.097158 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/simulator-w5-b1c718-kserve-6855bb954-k4w9w"] Oct 30 00:33:53.209453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.209367 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" path="/var/lib/kubelet/pods/dc58932a-ae73-4b75-862b-30ebbdebfeb4/volumes" Oct 30 00:33:53.403807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.403763 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:53.659972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.659933 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:33:53.660254 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:53.660225 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 00:33:54.677482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:54.677043 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:33:54.677482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:54.677113 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 00:33:55.101405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:55.101368 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:55.665239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:55.665205 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:55.665490 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:55.665471 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:55.666330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:55.666307 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:33:55.996017 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:55.995932 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:56.100399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:56.100362 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:57.055256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:57.055225 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:57.170487 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:57.160038 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:57.323556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:57.323525 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:57.790198 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:57.790097 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:57.905351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:57.905313 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:58.668716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.668678 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw"] Oct 30 00:33:58.669437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.669420 4433 cpu_manager.go:414] "containerMap: removing container" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" Oct 30 00:33:58.669487 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.669438 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" Oct 30 00:33:58.669618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.669593 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc58932a-ae73-4b75-862b-30ebbdebfeb4" containerName="main" Oct 30 00:33:58.675378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.675348 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.700245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.700211 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw"] Oct 30 00:33:58.704290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.702770 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-415d24-serving-cert\"" Oct 30 00:33:58.759642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.759537 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:58.767742 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.767680 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.770397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.770370 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfvs\" (UniqueName: \"kubernetes.io/projected/3b5cc49e-a720-4888-ab38-37f7380c3d5f-kube-api-access-jkfvs\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.770939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.770913 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.771130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.771116 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5cc49e-a720-4888-ab38-37f7380c3d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.868539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.868111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:58.872848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.872819 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5cc49e-a720-4888-ab38-37f7380c3d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.873012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.872963 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.873089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.873031 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfvs\" (UniqueName: \"kubernetes.io/projected/3b5cc49e-a720-4888-ab38-37f7380c3d5f-kube-api-access-jkfvs\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.873317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.873299 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.873685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.873660 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5cc49e-a720-4888-ab38-37f7380c3d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.875809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.875789 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.876023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.876001 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.897468 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.897438 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfvs\" (UniqueName: \"kubernetes.io/projected/3b5cc49e-a720-4888-ab38-37f7380c3d5f-kube-api-access-jkfvs\") pod \"maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:58.970271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:58.970188 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:59.004399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:59.004359 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:33:59.359829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:59.359798 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw"] Oct 30 00:33:59.360062 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:33:59.360034 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5cc49e_a720_4888_ab38_37f7380c3d5f.slice/crio-aaffc00d5e11021702aea58718acf689e22c9e6f2d59c0d1eb1db4e08fe8b2c5 WatchSource:0}: Error finding container aaffc00d5e11021702aea58718acf689e22c9e6f2d59c0d1eb1db4e08fe8b2c5: Status 404 returned error can't find the container with id aaffc00d5e11021702aea58718acf689e22c9e6f2d59c0d1eb1db4e08fe8b2c5 Oct 30 00:33:59.361532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:59.361515 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:33:59.921210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:33:59.916768 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:00.033607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:00.033464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:00.118485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:00.118452 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" event={"ID":"3b5cc49e-a720-4888-ab38-37f7380c3d5f","Type":"ContainerStarted","Data":"f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80"} Oct 30 00:34:00.118730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:00.118532 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" event={"ID":"3b5cc49e-a720-4888-ab38-37f7380c3d5f","Type":"ContainerStarted","Data":"aaffc00d5e11021702aea58718acf689e22c9e6f2d59c0d1eb1db4e08fe8b2c5"} Oct 30 00:34:00.118809 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:00.118763 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:34:00.163339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:00.162839 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" podStartSLOduration=1.789662067 podStartE2EDuration="2.162824158s" totalImagesPullingTime="373.162091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:00.162049485 +0000 UTC m=+1749.096118021" watchObservedRunningTime="2026-10-30 00:34:00.162824158 +0000 UTC m=+1749.096892692" Oct 30 00:34:00.618672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:00.618623 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:02.331073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:02.331040 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:04.506602 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:04.501937 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 00:34:04.675784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:04.675599 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:34:04.862602 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:04.801757 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:04.927876 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:04.927822 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:04.963597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:04.963171 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:34:05.147664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.120688 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.158161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.158109 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.177595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.160779 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:05.300598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.290743 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.392608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.354882 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.392608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.354824 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.392608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.372509 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.499063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.498952 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.515279 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.515239 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 00:34:05.534043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.532968 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:34:05.915729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.915665 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.927736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.927657 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:34:05.935621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.928346 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.972216 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.937484 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:34:05.975694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:05.975647 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:34:06.034390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.034351 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:34:06.040651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.001824 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.041315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.041289 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:34:06.121144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.121098 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 00:34:06.196666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.161552 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:34:06.730802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.370788 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.798477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.798422 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.373085 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.373150 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.627892 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.372081 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.851154 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.851217 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.851256 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.851290 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.851326 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:34:06.865367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.851365 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:34:06.928395 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.928350 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.969435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.948816 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.971038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.952381 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" containerID="cri-o://618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7" gracePeriod=45 Oct 30 00:34:06.972917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:06.972886 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 00:34:07.070836 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.067664 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:34:07.223606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.221773 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:34:07.223606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.221836 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:07.223606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.222527 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:07.366471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.366347 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" containerID="cri-o://9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937" gracePeriod=44 Oct 30 00:34:07.772674 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.761788 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:34:07.804597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.804091 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:07.904460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.904420 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 00:34:07.905832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.905802 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 00:34:07.907227 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.907205 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" podUID="de398e8d-5cd0-4e2f-ae60-7bf17a4a942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 00:34:07.907890 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.907868 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 00:34:07.909052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.909031 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:34:07.909741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.909718 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 00:34:07.918104 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:07.918079 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:34:07.926719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.926692 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused" Oct 30 00:34:07.967612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.966793 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 00:34:07.970827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:07.970798 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:34:08.075646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.070038 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 00:34:08.094934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.086205 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:34:08.094934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.086269 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:34:08.094934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.086322 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:34:08.094934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.086365 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:34:08.094934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.086412 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v" Oct 30 00:34:08.094934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.086453 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:34:08.094934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.086495 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:34:08.105748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.096478 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:08.120849 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.120812 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": EOF" Oct 30 00:34:08.228362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.227836 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:08.255608 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.253659 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:34:08.276795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.276764 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:34:08.280598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.279207 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused" Oct 30 00:34:08.446445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.446418 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:34:08.558270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.558234 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-dshm\") pod \"d377874c-82d4-452e-af77-fdfc2a90fc76\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " Oct 30 00:34:08.558457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.558413 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d377874c-82d4-452e-af77-fdfc2a90fc76-tls-certs\" (UniqueName: \"kubernetes.io/secret/d377874c-82d4-452e-af77-fdfc2a90fc76-tls-certs\") pod \"d377874c-82d4-452e-af77-fdfc2a90fc76\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " Oct 30 00:34:08.558531 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.558465 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-model-cache\") pod \"d377874c-82d4-452e-af77-fdfc2a90fc76\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " Oct 30 00:34:08.558603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.558546 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-home\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-home\") pod \"d377874c-82d4-452e-af77-fdfc2a90fc76\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " Oct 30 00:34:08.558679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.558645 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-tmp-dir\") pod \"d377874c-82d4-452e-af77-fdfc2a90fc76\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " Oct 30 00:34:08.558805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.558791 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d377874c-82d4-452e-af77-fdfc2a90fc76-kube-api-access-dj7cc\" (UniqueName: \"kubernetes.io/projected/d377874c-82d4-452e-af77-fdfc2a90fc76-kube-api-access-dj7cc\") pod \"d377874c-82d4-452e-af77-fdfc2a90fc76\" (UID: \"d377874c-82d4-452e-af77-fdfc2a90fc76\") " Oct 30 00:34:08.559366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.559001 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-model-cache" pod "d377874c-82d4-452e-af77-fdfc2a90fc76" (UID: "d377874c-82d4-452e-af77-fdfc2a90fc76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.559366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.559124 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-tmp-dir" pod "d377874c-82d4-452e-af77-fdfc2a90fc76" (UID: "d377874c-82d4-452e-af77-fdfc2a90fc76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.559366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.559280 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-home" pod "d377874c-82d4-452e-af77-fdfc2a90fc76" (UID: "d377874c-82d4-452e-af77-fdfc2a90fc76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.559738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.559614 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.559738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.559636 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.559738 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.559649 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.561636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.561566 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-dshm" pod "d377874c-82d4-452e-af77-fdfc2a90fc76" (UID: "d377874c-82d4-452e-af77-fdfc2a90fc76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.561787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.561720 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d377874c-82d4-452e-af77-fdfc2a90fc76-tls-certs" pod "d377874c-82d4-452e-af77-fdfc2a90fc76" (UID: "d377874c-82d4-452e-af77-fdfc2a90fc76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:08.561787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.561738 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d377874c-82d4-452e-af77-fdfc2a90fc76-kube-api-access-dj7cc" pod "d377874c-82d4-452e-af77-fdfc2a90fc76" (UID: "d377874c-82d4-452e-af77-fdfc2a90fc76"). InnerVolumeSpecName "kube-api-access-dj7cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:08.661841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.661759 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377874c-82d4-452e-af77-fdfc2a90fc76-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.661841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.661790 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj7cc\" (UniqueName: \"kubernetes.io/projected/d377874c-82d4-452e-af77-fdfc2a90fc76-kube-api-access-dj7cc\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.661841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.661801 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377874c-82d4-452e-af77-fdfc2a90fc76-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.688604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.688565 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:34:08.864206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.864172 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-tmp-dir\") pod \"10585d8e-dac6-4873-a39c-86ca0525f1b6\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " Oct 30 00:34:08.864755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.864347 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10585d8e-dac6-4873-a39c-86ca0525f1b6-kube-api-access-r5bxh\" (UniqueName: \"kubernetes.io/projected/10585d8e-dac6-4873-a39c-86ca0525f1b6-kube-api-access-r5bxh\") pod \"10585d8e-dac6-4873-a39c-86ca0525f1b6\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " Oct 30 00:34:08.864755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.864417 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10585d8e-dac6-4873-a39c-86ca0525f1b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/10585d8e-dac6-4873-a39c-86ca0525f1b6-tls-certs\") pod \"10585d8e-dac6-4873-a39c-86ca0525f1b6\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " Oct 30 00:34:08.864755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.864562 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-model-cache\") pod \"10585d8e-dac6-4873-a39c-86ca0525f1b6\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " Oct 30 00:34:08.864755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.864626 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-dshm\") pod \"10585d8e-dac6-4873-a39c-86ca0525f1b6\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " Oct 30 00:34:08.864755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.864708 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-tmp-dir" pod "10585d8e-dac6-4873-a39c-86ca0525f1b6" (UID: "10585d8e-dac6-4873-a39c-86ca0525f1b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.864996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.864777 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-home\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-home\") pod \"10585d8e-dac6-4873-a39c-86ca0525f1b6\" (UID: \"10585d8e-dac6-4873-a39c-86ca0525f1b6\") " Oct 30 00:34:08.865083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.865060 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-home" pod "10585d8e-dac6-4873-a39c-86ca0525f1b6" (UID: "10585d8e-dac6-4873-a39c-86ca0525f1b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.865140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.865072 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-model-cache" pod "10585d8e-dac6-4873-a39c-86ca0525f1b6" (UID: "10585d8e-dac6-4873-a39c-86ca0525f1b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.865474 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.865453 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.865474 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.865471 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.865689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.865482 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.866741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.866719 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10585d8e-dac6-4873-a39c-86ca0525f1b6-tls-certs" pod "10585d8e-dac6-4873-a39c-86ca0525f1b6" (UID: "10585d8e-dac6-4873-a39c-86ca0525f1b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:08.866741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.866731 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10585d8e-dac6-4873-a39c-86ca0525f1b6-kube-api-access-r5bxh" pod "10585d8e-dac6-4873-a39c-86ca0525f1b6" (UID: "10585d8e-dac6-4873-a39c-86ca0525f1b6"). InnerVolumeSpecName "kube-api-access-r5bxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:08.866873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.866753 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-dshm" pod "10585d8e-dac6-4873-a39c-86ca0525f1b6" (UID: "10585d8e-dac6-4873-a39c-86ca0525f1b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:08.966759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.966730 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5bxh\" (UniqueName: \"kubernetes.io/projected/10585d8e-dac6-4873-a39c-86ca0525f1b6-kube-api-access-r5bxh\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.966759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.966758 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10585d8e-dac6-4873-a39c-86ca0525f1b6-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.966939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.966770 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10585d8e-dac6-4873-a39c-86ca0525f1b6-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.981172 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.981145 4433 generic.go:356] "Generic (PLEG): container finished" podID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerID="618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7" exitCode=0 Oct 30 00:34:08.981310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.981201 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" Oct 30 00:34:08.981310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.981233 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" event={"ID":"d377874c-82d4-452e-af77-fdfc2a90fc76","Type":"ContainerDied","Data":"618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7"} Oct 30 00:34:08.981310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.981293 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84" event={"ID":"d377874c-82d4-452e-af77-fdfc2a90fc76","Type":"ContainerDied","Data":"8b6e8c6bb194bc75fedb465a53b8b74ae4f5c209af9a705ea6b64e83dacf018e"} Oct 30 00:34:08.981480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.981320 4433 scope.go:122] "RemoveContainer" containerID="618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7" Oct 30 00:34:08.982979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.982955 4433 generic.go:356] "Generic (PLEG): container finished" podID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerID="9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937" exitCode=0 Oct 30 00:34:08.983104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.983013 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" Oct 30 00:34:08.983104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.983031 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" event={"ID":"10585d8e-dac6-4873-a39c-86ca0525f1b6","Type":"ContainerDied","Data":"9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937"} Oct 30 00:34:08.983104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.983056 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9" event={"ID":"10585d8e-dac6-4873-a39c-86ca0525f1b6","Type":"ContainerDied","Data":"b31d838109659e65b5d8f745f1d44a28ff0c8386e1b2eb50e894a2242a8037c4"} Oct 30 00:34:08.993109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.993087 4433 scope.go:122] "RemoveContainer" containerID="618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7" Oct 30 00:34:08.993437 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:08.993409 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7\": container with ID starting with 618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7 not found: ID does not exist" containerID="618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7" Oct 30 00:34:08.993696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.993445 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7"} err="failed to get container status \"618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7\": rpc error: code = NotFound desc = could not find container \"618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7\": container with ID starting with 618c9077576cb3e557eab30a7bb6be5d62866a6414bd2f87d375927c3bef73b7 not found: ID does not exist" Oct 30 00:34:08.993696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:08.993466 4433 scope.go:122] "RemoveContainer" containerID="9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937" Oct 30 00:34:09.003564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.003543 4433 scope.go:122] "RemoveContainer" containerID="9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937" Oct 30 00:34:09.003954 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:09.003934 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937\": container with ID starting with 9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937 not found: ID does not exist" containerID="9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937" Oct 30 00:34:09.004018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.003959 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937"} err="failed to get container status \"9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937\": rpc error: code = NotFound desc = could not find container \"9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937\": container with ID starting with 9fc6604761c3b9a27d69bf678e7829009ecee313f2ff042b4b23158ec9a1c937 not found: ID does not exist" Oct 30 00:34:09.036689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.036659 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9"] Oct 30 00:34:09.040567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.040543 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/simulator-w3-564c8f-kserve-7fd69bcb4b-fbtf9"] Oct 30 00:34:09.075059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.074783 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84"] Oct 30 00:34:09.080210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.080188 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-2-w3-564c8f-kserve-558b5f487c-5nw84"] Oct 30 00:34:09.181399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.181368 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" path="/var/lib/kubelet/pods/10585d8e-dac6-4873-a39c-86ca0525f1b6/volumes" Oct 30 00:34:09.181898 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.181883 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" path="/var/lib/kubelet/pods/d377874c-82d4-452e-af77-fdfc2a90fc76/volumes" Oct 30 00:34:09.786873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.786834 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm"] Oct 30 00:34:09.787625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.787583 4433 cpu_manager.go:414] "containerMap: removing container" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" Oct 30 00:34:09.787625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.787601 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" Oct 30 00:34:09.787812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.787642 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" Oct 30 00:34:09.787812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.787647 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" Oct 30 00:34:09.787812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.787763 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d377874c-82d4-452e-af77-fdfc2a90fc76" containerName="main" Oct 30 00:34:09.787812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.787775 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="10585d8e-dac6-4873-a39c-86ca0525f1b6" containerName="main" Oct 30 00:34:09.793247 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.793222 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.831741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.816361 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm"] Oct 30 00:34:09.831741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.819725 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-05c6b2-serving-cert\"" Oct 30 00:34:09.891410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.891366 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:09.893861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.893829 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.894014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.893913 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhmt\" (UniqueName: \"kubernetes.io/projected/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-kube-api-access-xfhmt\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.894099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.894083 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.894163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.894124 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.996673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.996638 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.997003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.996971 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhmt\" (UniqueName: \"kubernetes.io/projected/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-kube-api-access-xfhmt\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.997313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.997293 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.997458 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.997439 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:09.998207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:09.998167 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:10.000418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:10.000387 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:10.004703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:10.004675 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:10.010326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:10.010300 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:10.022146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:10.022068 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhmt\" (UniqueName: \"kubernetes.io/projected/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-kube-api-access-xfhmt\") pod \"maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:10.110195 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:10.110162 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:10.119515 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:10.119484 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:10.437410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:10.437375 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm"] Oct 30 00:34:10.438262 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:34:10.438237 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bbbf0e_a33c_40b1_82bd_7a73bb9699ad.slice/crio-1d54a015cd17cb8aa2757ce627ed6d048df12e281aa10ce3eb1315fd8f59d0d7 WatchSource:0}: Error finding container 1d54a015cd17cb8aa2757ce627ed6d048df12e281aa10ce3eb1315fd8f59d0d7: Status 404 returned error can't find the container with id 1d54a015cd17cb8aa2757ce627ed6d048df12e281aa10ce3eb1315fd8f59d0d7 Oct 30 00:34:11.068633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:11.068483 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" event={"ID":"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad","Type":"ContainerStarted","Data":"3bdf7da0a3b08fbf26fd00c4404e495d78bd3eb4b279ed52e22d1deebef6aca6"} Oct 30 00:34:11.068633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:11.068547 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" event={"ID":"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad","Type":"ContainerStarted","Data":"1d54a015cd17cb8aa2757ce627ed6d048df12e281aa10ce3eb1315fd8f59d0d7"} Oct 30 00:34:11.069714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:11.069688 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:11.115825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:11.115767 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" podStartSLOduration=1.7418662980000001 podStartE2EDuration="2.115751707s" totalImagesPullingTime="373.885409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:11.112829836 +0000 UTC m=+1760.046898392" watchObservedRunningTime="2026-10-30 00:34:11.115751707 +0000 UTC m=+1760.049820286" Oct 30 00:34:11.482064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:11.481637 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:11.583997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:11.583959 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:11.777656 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:11.777549 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:12.855058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:12.855028 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:13.553399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:13.553359 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:13.663668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:13.663631 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:15.269315 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:15.269255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:16.682891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:16.682855 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:18.097534 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:18.097501 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:34:18.287139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:18.287091 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:19.301074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:19.301038 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:19.785712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:19.785679 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:21.231446 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:21.231408 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:21.335077 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:21.335038 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:22.872764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:22.872734 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:24.144663 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.144602 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk"] Oct 30 00:34:24.145177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.145086 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" podUID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" containerName="maas-api" containerID="cri-o://297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e" gracePeriod=30 Oct 30 00:34:24.205804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.205765 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:24.595688 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.594914 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:34:24.696959 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.696924 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:24.729232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.729191 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls\") pod \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " Oct 30 00:34:24.729441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.729281 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-openshift-service-ca\") pod \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " Oct 30 00:34:24.729528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.729490 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-metrics-certs\") pod \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " Oct 30 00:34:24.729668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.729619 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-kube-api-access-lh6r6\" (UniqueName: \"kubernetes.io/projected/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-kube-api-access-lh6r6\") pod \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\" (UID: \"a98846a5-27d4-4a29-82ce-a573f0a1a3eb\") " Oct 30 00:34:24.729880 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.729856 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-openshift-service-ca" pod "a98846a5-27d4-4a29-82ce-a573f0a1a3eb" (UID: "a98846a5-27d4-4a29-82ce-a573f0a1a3eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:24.730587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.730537 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:24.732264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.732243 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls" pod "a98846a5-27d4-4a29-82ce-a573f0a1a3eb" (UID: "a98846a5-27d4-4a29-82ce-a573f0a1a3eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:24.733530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.733510 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-metrics-certs" pod "a98846a5-27d4-4a29-82ce-a573f0a1a3eb" (UID: "a98846a5-27d4-4a29-82ce-a573f0a1a3eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:24.734141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.734118 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-kube-api-access-lh6r6" pod "a98846a5-27d4-4a29-82ce-a573f0a1a3eb" (UID: "a98846a5-27d4-4a29-82ce-a573f0a1a3eb"). InnerVolumeSpecName "kube-api-access-lh6r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:24.809680 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.809638 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:24.832273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.832230 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:24.832273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.832266 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6r6\" (UniqueName: \"kubernetes.io/projected/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-kube-api-access-lh6r6\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:24.832273 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.832279 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98846a5-27d4-4a29-82ce-a573f0a1a3eb-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:24.930819 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:24.930779 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:25.197105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.195749 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" event={"ID":"a98846a5-27d4-4a29-82ce-a573f0a1a3eb","Type":"ContainerDied","Data":"297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e"} Oct 30 00:34:25.197105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.195853 4433 scope.go:122] "RemoveContainer" containerID="297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e" Oct 30 00:34:25.197105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.195762 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" Oct 30 00:34:25.197105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.195783 4433 generic.go:356] "Generic (PLEG): container finished" podID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" containerID="297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e" exitCode=0 Oct 30 00:34:25.197105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.196232 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" event={"ID":"a98846a5-27d4-4a29-82ce-a573f0a1a3eb","Type":"ContainerDied","Data":"2463d3a13cd42c2f3d803dbc42cbeba6b54032a4f41d148e520b95d494dd9010"} Oct 30 00:34:25.222043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.222022 4433 scope.go:122] "RemoveContainer" containerID="297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e" Oct 30 00:34:25.222380 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:25.222356 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e\": container with ID starting with 297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e not found: ID does not exist" containerID="297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e" Oct 30 00:34:25.222429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.222390 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e"} err="failed to get container status \"297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e\": rpc error: code = NotFound desc = could not find container \"297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e\": container with ID starting with 297eb960a9190b5aa7e1421dac0f7a41db08f3cc02bf7bbbc5f415c4d83e216e not found: ID does not exist" Oct 30 00:34:25.263922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.263881 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk"] Oct 30 00:34:25.276506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.276470 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk"] Oct 30 00:34:25.448109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:25.448019 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:26.142931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:26.142885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:26.818627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:26.817076 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:27.371595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:27.371339 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" path="/var/lib/kubelet/pods/a98846a5-27d4-4a29-82ce-a573f0a1a3eb/volumes" Oct 30 00:34:27.571596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:27.570722 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-564c8f-5769899c9c-csgfk" podUID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:29.080594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:29.068215 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.390448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:29.390354 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.524482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:29.524449 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.820026 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:29.819990 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:30.079201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:30.079101 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:30.639855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:30.639804 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:31.304082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:31.304046 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:31.682274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:31.682237 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:32.093462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.093270 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:32.204610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.204540 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:32.471654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.456910 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v"] Oct 30 00:34:32.472153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.472111 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" containerID="cri-o://eb146ad4fb71b13f40c26536c680c9147ecc7a4a36aa39dd9abee3f801a4e1af" gracePeriod=30 Oct 30 00:34:32.497025 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.496997 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:32.497234 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.497220 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng"] Oct 30 00:34:32.497556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.497532 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" containerID="cri-o://05c0b6e780b638933a6a60865cded6bb280532686d74a304aaed87f216b0134e" gracePeriod=30 Oct 30 00:34:32.506966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.506941 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf"] Oct 30 00:34:32.507178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.507158 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" containerID="cri-o://055c0a9252aa64232e83a07b83dfbf0bfa1d76ebfb07d9182c213135e2e3f0e4" gracePeriod=30 Oct 30 00:34:32.511919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.511901 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv"] Oct 30 00:34:32.512099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.512082 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" containerID="cri-o://70dc0174219e9d532565fd16c872c784c826defaef469901ca4a7b426605ee16" gracePeriod=30 Oct 30 00:34:32.519106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.519005 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:32.519106 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.519052 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:32.737897 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.737797 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:32.858146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:32.858095 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:33.408008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:33.407971 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.088670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:34.088629 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:34.089140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:34.088711 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:34.653059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:34.653019 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps"] Oct 30 00:34:34.653281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:34.653252 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" podUID="6f1ef643-a2f0-4cd2-8788-08b8084b0dca" containerName="maas-api" containerID="cri-o://2ad4b50f0bdaabe141add6aa6357f1a23d0caab451c4fcfb46f7bb83bd562043" gracePeriod=30 Oct 30 00:34:34.713937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:34.713876 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.821382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:34.821347 4433 generic.go:356] "Generic (PLEG): container finished" podID="6f1ef643-a2f0-4cd2-8788-08b8084b0dca" containerID="2ad4b50f0bdaabe141add6aa6357f1a23d0caab451c4fcfb46f7bb83bd562043" exitCode=0 Oct 30 00:34:34.821568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:34.821422 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" event={"ID":"6f1ef643-a2f0-4cd2-8788-08b8084b0dca","Type":"ContainerDied","Data":"2ad4b50f0bdaabe141add6aa6357f1a23d0caab451c4fcfb46f7bb83bd562043"} Oct 30 00:34:35.008721 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.007150 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:34:35.118954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.118903 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-maas-api-tls\") pod \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " Oct 30 00:34:35.119431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.119024 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-openshift-service-ca\") pod \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " Oct 30 00:34:35.119431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.119061 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-kube-api-access-m8khg\" (UniqueName: \"kubernetes.io/projected/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-kube-api-access-m8khg\") pod \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " Oct 30 00:34:35.119431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.119129 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-metrics-certs\") pod \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\" (UID: \"6f1ef643-a2f0-4cd2-8788-08b8084b0dca\") " Oct 30 00:34:35.119700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.119479 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-openshift-service-ca" pod "6f1ef643-a2f0-4cd2-8788-08b8084b0dca" (UID: "6f1ef643-a2f0-4cd2-8788-08b8084b0dca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:35.120068 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.120040 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:35.124382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.122977 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-maas-api-tls" pod "6f1ef643-a2f0-4cd2-8788-08b8084b0dca" (UID: "6f1ef643-a2f0-4cd2-8788-08b8084b0dca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:35.124382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.122991 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-kube-api-access-m8khg" pod "6f1ef643-a2f0-4cd2-8788-08b8084b0dca" (UID: "6f1ef643-a2f0-4cd2-8788-08b8084b0dca"). InnerVolumeSpecName "kube-api-access-m8khg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:35.129218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.124807 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-metrics-certs" pod "6f1ef643-a2f0-4cd2-8788-08b8084b0dca" (UID: "6f1ef643-a2f0-4cd2-8788-08b8084b0dca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:35.221958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.221866 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:35.221958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.221903 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8khg\" (UniqueName: \"kubernetes.io/projected/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-kube-api-access-m8khg\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:35.221958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.221918 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1ef643-a2f0-4cd2-8788-08b8084b0dca-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:35.277377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.277340 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:35.395777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.395721 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:35.508930 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.508816 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:35.831321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.831289 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" Oct 30 00:34:35.831321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.831312 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps" event={"ID":"6f1ef643-a2f0-4cd2-8788-08b8084b0dca","Type":"ContainerDied","Data":"3e6fdd8b0c4699b4a17bd65387d73b9bde16a0bbb492cdc2ebdc963280e108ee"} Oct 30 00:34:35.831615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.831374 4433 scope.go:122] "RemoveContainer" containerID="2ad4b50f0bdaabe141add6aa6357f1a23d0caab451c4fcfb46f7bb83bd562043" Oct 30 00:34:35.888389 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.888355 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps"] Oct 30 00:34:35.901672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.901640 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b1c718-68bff76f8f-n4gps"] Oct 30 00:34:35.961631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.961596 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:35.962641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.962613 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:35.962791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:35.962670 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:36.592802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:36.592759 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:36.593207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:36.592828 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:36.961851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:36.961775 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn"] Oct 30 00:34:36.962026 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:36.961993 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" containerID="cri-o://79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453" gracePeriod=30 Oct 30 00:34:37.092342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.092296 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:37.105485 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:37.105440 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:34:37.183181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.183143 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1ef643-a2f0-4cd2-8788-08b8084b0dca" path="/var/lib/kubelet/pods/6f1ef643-a2f0-4cd2-8788-08b8084b0dca/volumes" Oct 30 00:34:37.858246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.856599 4433 generic.go:356] "Generic (PLEG): container finished" podID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerID="05c0b6e780b638933a6a60865cded6bb280532686d74a304aaed87f216b0134e" exitCode=0 Oct 30 00:34:37.858246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.856799 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" event={"ID":"2186b19c-57bf-41cb-bf4e-a9800a8e2934","Type":"ContainerDied","Data":"05c0b6e780b638933a6a60865cded6bb280532686d74a304aaed87f216b0134e"} Oct 30 00:34:37.860248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.860221 4433 generic.go:356] "Generic (PLEG): container finished" podID="78b75034-6711-4f76-9105-32372500328c" containerID="055c0a9252aa64232e83a07b83dfbf0bfa1d76ebfb07d9182c213135e2e3f0e4" exitCode=0 Oct 30 00:34:37.860387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.860334 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" event={"ID":"78b75034-6711-4f76-9105-32372500328c","Type":"ContainerDied","Data":"055c0a9252aa64232e83a07b83dfbf0bfa1d76ebfb07d9182c213135e2e3f0e4"} Oct 30 00:34:37.860387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.860369 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" event={"ID":"78b75034-6711-4f76-9105-32372500328c","Type":"ContainerDied","Data":"0d9a890fcdb8ecfa0301547b73ca003f57cbf952d2013cdc607a23a20c30b0d8"} Oct 30 00:34:37.860523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.860391 4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d9a890fcdb8ecfa0301547b73ca003f57cbf952d2013cdc607a23a20c30b0d8" Oct 30 00:34:37.862347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.862308 4433 generic.go:356] "Generic (PLEG): container finished" podID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerID="70dc0174219e9d532565fd16c872c784c826defaef469901ca4a7b426605ee16" exitCode=0 Oct 30 00:34:37.862508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.862427 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" event={"ID":"f310099f-ae88-4cfe-9ee1-3175b696ab12","Type":"ContainerDied","Data":"70dc0174219e9d532565fd16c872c784c826defaef469901ca4a7b426605ee16"} Oct 30 00:34:37.864769 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.864743 4433 generic.go:356] "Generic (PLEG): container finished" podID="eb4612bf-5599-477a-9abe-c2a15751d372" containerID="eb146ad4fb71b13f40c26536c680c9147ecc7a4a36aa39dd9abee3f801a4e1af" exitCode=0 Oct 30 00:34:37.864871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.864799 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" event={"ID":"eb4612bf-5599-477a-9abe-c2a15751d372","Type":"ContainerDied","Data":"eb146ad4fb71b13f40c26536c680c9147ecc7a4a36aa39dd9abee3f801a4e1af"} Oct 30 00:34:37.880302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.880270 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:34:37.918212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.918172 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:34:37.922779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:37.922756 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:34:38.016627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016511 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b75034-6711-4f76-9105-32372500328c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b75034-6711-4f76-9105-32372500328c-istiod-ca-cert\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.016627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016620 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-credential-socket\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.016810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016677 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-token\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-token\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.016810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016765 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-data\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.016931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016800 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-socket\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.016931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016843 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78b75034-6711-4f76-9105-32372500328c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b75034-6711-4f76-9105-32372500328c-istio-podinfo\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.016931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016882 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-token\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-token\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.016975 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-envoy\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.017085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017005 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-istio-token\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-istio-token\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.017085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017050 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-kube-api-access-v6r6l\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-kube-api-access-v6r6l\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017103 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-data\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.017245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017164 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-podinfo\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017191 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f310099f-ae88-4cfe-9ee1-3175b696ab12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f310099f-ae88-4cfe-9ee1-3175b696ab12-istiod-ca-cert\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017277 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-kube-api-access-wk4h7\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-kube-api-access-wk4h7\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.017399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017325 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-credential-socket\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017401 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-certs\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.017679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017497 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-envoy\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017642 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-podinfo\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.017679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017674 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-socket\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017762 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istiod-ca-cert\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.017918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017792 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-credential-socket\") pod \"78b75034-6711-4f76-9105-32372500328c\" (UID: \"78b75034-6711-4f76-9105-32372500328c\") " Oct 30 00:34:38.017918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017830 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-socket\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.017918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017855 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-data\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.017918 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017905 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-certs\") pod \"f310099f-ae88-4cfe-9ee1-3175b696ab12\" (UID: \"f310099f-ae88-4cfe-9ee1-3175b696ab12\") " Oct 30 00:34:38.018169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.017971 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-envoy\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.018169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.018006 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-kube-api-access-hqrgx\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-kube-api-access-hqrgx\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.018169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.018078 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-certs\") pod \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\" (UID: \"2186b19c-57bf-41cb-bf4e-a9800a8e2934\") " Oct 30 00:34:38.019763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.019112 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f310099f-ae88-4cfe-9ee1-3175b696ab12-istiod-ca-cert" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:38.019763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.019207 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istiod-ca-cert" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:38.019763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.019416 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-credential-socket" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.019763 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.019721 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-socket" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.020020 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.019906 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-data" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.021697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.020092 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-certs" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.021697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.020713 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-data" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.021697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.020955 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b75034-6711-4f76-9105-32372500328c-istiod-ca-cert" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:38.021697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.021191 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-credential-socket" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.021697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.021442 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-socket" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.023144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.022893 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-kube-api-access-hqrgx" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "kube-api-access-hqrgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.024466 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-envoy" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.024501 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-envoy" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.024765 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-certs" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.024792 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-kube-api-access-v6r6l" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "kube-api-access-v6r6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.024812 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-credential-socket" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.024979 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-socket" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.025008 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-certs" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.025271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.025155 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-data" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.025658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.025638 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-token" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.027054 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.027017 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-podinfo" pod "2186b19c-57bf-41cb-bf4e-a9800a8e2934" (UID: "2186b19c-57bf-41cb-bf4e-a9800a8e2934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:38.027430 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.027400 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-podinfo" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:38.027922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.027893 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-envoy" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.028135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.028114 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b75034-6711-4f76-9105-32372500328c-istio-podinfo" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:38.028249 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.028230 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-token" pod "f310099f-ae88-4cfe-9ee1-3175b696ab12" (UID: "f310099f-ae88-4cfe-9ee1-3175b696ab12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.030079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.030051 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-kube-api-access-wk4h7" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "kube-api-access-wk4h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.030439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.030416 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-istio-token" pod "78b75034-6711-4f76-9105-32372500328c" (UID: "78b75034-6711-4f76-9105-32372500328c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.121418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121375 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121419 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121435 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121449 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121464 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121477 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121490 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121503 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121514 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121526 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqrgx\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-kube-api-access-hqrgx\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121539 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121553 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b75034-6711-4f76-9105-32372500328c-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121565 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121602 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121618 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121630 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.121641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121643 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b75034-6711-4f76-9105-32372500328c-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121657 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121670 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121682 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121697 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6r6l\" (UniqueName: \"kubernetes.io/projected/f310099f-ae88-4cfe-9ee1-3175b696ab12-kube-api-access-v6r6l\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121710 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2186b19c-57bf-41cb-bf4e-a9800a8e2934-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121725 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f310099f-ae88-4cfe-9ee1-3175b696ab12-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121737 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f310099f-ae88-4cfe-9ee1-3175b696ab12-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121749 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk4h7\" (UniqueName: \"kubernetes.io/projected/78b75034-6711-4f76-9105-32372500328c-kube-api-access-wk4h7\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121763 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f310099f-ae88-4cfe-9ee1-3175b696ab12-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.122121 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.121776 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b75034-6711-4f76-9105-32372500328c-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.211772 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.211720 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:34:38.325115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325076 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-envoy\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325127 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-certs\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325159 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4612bf-5599-477a-9abe-c2a15751d372-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4612bf-5599-477a-9abe-c2a15751d372-istiod-ca-cert\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325266 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-socket\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325331 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-istio-token\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-istio-token\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325436 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325430 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb4612bf-5599-477a-9abe-c2a15751d372-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4612bf-5599-477a-9abe-c2a15751d372-istio-podinfo\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325424 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-certs" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.325556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325519 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-kube-api-access-ggr5v\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-kube-api-access-ggr5v\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325590 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-data\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325664 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-socket" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.325882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325692 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4612bf-5599-477a-9abe-c2a15751d372-istiod-ca-cert" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:38.325882 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325695 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-credential-socket\") pod \"eb4612bf-5599-477a-9abe-c2a15751d372\" (UID: \"eb4612bf-5599-477a-9abe-c2a15751d372\") " Oct 30 00:34:38.325986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325893 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-credential-socket" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.325986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.325946 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-data" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.326673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.326599 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.326673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.326626 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb4612bf-5599-477a-9abe-c2a15751d372-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.326673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.326638 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.326673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.326649 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.326673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.326662 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.328003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.327966 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-istio-token" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.328108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.328020 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-kube-api-access-ggr5v" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "kube-api-access-ggr5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.328108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.328028 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-envoy" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.328473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.328453 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb4612bf-5599-477a-9abe-c2a15751d372-istio-podinfo" pod "eb4612bf-5599-477a-9abe-c2a15751d372" (UID: "eb4612bf-5599-477a-9abe-c2a15751d372"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:38.428450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.428413 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb4612bf-5599-477a-9abe-c2a15751d372-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.428450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.428447 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.429642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.428461 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb4612bf-5599-477a-9abe-c2a15751d372-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.429642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.428474 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggr5v\" (UniqueName: \"kubernetes.io/projected/eb4612bf-5599-477a-9abe-c2a15751d372-kube-api-access-ggr5v\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.791152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.791116 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:38.878825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.878793 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" Oct 30 00:34:38.879647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.878812 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv" event={"ID":"f310099f-ae88-4cfe-9ee1-3175b696ab12","Type":"ContainerDied","Data":"2195d4b3182f7a11adc63ae171e67824d710dfa85d0c03b56824d7b4d1a2ff8b"} Oct 30 00:34:38.879647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.878893 4433 scope.go:122] "RemoveContainer" containerID="70dc0174219e9d532565fd16c872c784c826defaef469901ca4a7b426605ee16" Oct 30 00:34:38.881487 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.881461 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" event={"ID":"eb4612bf-5599-477a-9abe-c2a15751d372","Type":"ContainerDied","Data":"f15c62b287bd2af74d1eec21f2f002663ee1ba9b8ad4f83af04b2a8e7e0c23ad"} Oct 30 00:34:38.881715 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.881473 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v" Oct 30 00:34:38.883468 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.883445 4433 generic.go:356] "Generic (PLEG): container finished" podID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerID="7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51" exitCode=0 Oct 30 00:34:38.883606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.883523 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" event={"ID":"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499","Type":"ContainerDied","Data":"7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51"} Oct 30 00:34:38.884135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.884075 4433 scope.go:122] "RemoveContainer" containerID="7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51" Oct 30 00:34:38.885181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.885165 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf" Oct 30 00:34:38.886000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.885979 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" Oct 30 00:34:38.887649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.887628 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng" event={"ID":"2186b19c-57bf-41cb-bf4e-a9800a8e2934","Type":"ContainerDied","Data":"893f673bf8ea922b3ae3c578fbdab39fb2ef21ebf595873732c14605d9814208"} Oct 30 00:34:38.889053 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.889015 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:34:38.889181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.889111 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:34:38.889181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.889161 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:34:38.895169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.895110 4433 scope.go:122] "RemoveContainer" containerID="eb146ad4fb71b13f40c26536c680c9147ecc7a4a36aa39dd9abee3f801a4e1af" Oct 30 00:34:38.909508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:38.909482 4433 scope.go:122] "RemoveContainer" containerID="05c0b6e780b638933a6a60865cded6bb280532686d74a304aaed87f216b0134e" Oct 30 00:34:39.034346 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.034319 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv"] Oct 30 00:34:39.040079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.040049 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-5hsgv"] Oct 30 00:34:39.070016 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.069979 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v"] Oct 30 00:34:39.073566 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.073546 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-wr28v"] Oct 30 00:34:39.113941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.113906 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng"] Oct 30 00:34:39.117734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.117705 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-fb9ng"] Oct 30 00:34:39.151393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.151316 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf"] Oct 30 00:34:39.157970 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.157940 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-dkqvf"] Oct 30 00:34:39.181357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.181320 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" path="/var/lib/kubelet/pods/2186b19c-57bf-41cb-bf4e-a9800a8e2934/volumes" Oct 30 00:34:39.181889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.181874 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b75034-6711-4f76-9105-32372500328c" path="/var/lib/kubelet/pods/78b75034-6711-4f76-9105-32372500328c/volumes" Oct 30 00:34:39.182335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.182322 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" path="/var/lib/kubelet/pods/eb4612bf-5599-477a-9abe-c2a15751d372/volumes" Oct 30 00:34:39.183186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.183168 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" path="/var/lib/kubelet/pods/f310099f-ae88-4cfe-9ee1-3175b696ab12/volumes" Oct 30 00:34:39.803357 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.803324 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:39.896182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:39.896148 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" event={"ID":"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499","Type":"ContainerStarted","Data":"6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f"} Oct 30 00:34:40.377818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:40.377780 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:34:40.379264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:40.379233 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 00:34:40.379404 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:40.379293 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 00:34:40.412213 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:40.412184 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:40.519475 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:40.519435 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:40.630262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:40.629032 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:41.378309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:41.378270 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 00:34:41.378751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:41.378326 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 00:34:42.212235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.212098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:42.392841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.388116 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 00:34:42.392841 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.388196 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 00:34:42.445066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.445037 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:34:42.624688 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.624653 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbe97ca-af94-4472-9fd2-87ea421bd627-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe97ca-af94-4472-9fd2-87ea421bd627-istiod-ca-cert\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.624846 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.624708 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-envoy\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.624846 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.624822 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-data\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.624965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.624881 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-certs\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.625015 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.624959 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-podinfo\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.625155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625110 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-data" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:42.625241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625163 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-certs" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:42.625241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625194 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbe97ca-af94-4472-9fd2-87ea421bd627-istiod-ca-cert" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:42.625360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625322 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-kube-api-access-zkvxc\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-kube-api-access-zkvxc\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.625360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625359 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-socket\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.625477 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625430 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-token\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.625635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625607 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-credential-socket\") pod \"6bbe97ca-af94-4472-9fd2-87ea421bd627\" (UID: \"6bbe97ca-af94-4472-9fd2-87ea421bd627\") " Oct 30 00:34:42.625797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.625771 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-socket" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:42.626120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.626096 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-credential-socket" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:42.626508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.626485 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe97ca-af94-4472-9fd2-87ea421bd627-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.626629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.626512 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.626629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.626527 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.626629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.626540 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.626629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.626554 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.627372 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.627342 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-podinfo" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:42.627852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.627814 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-kube-api-access-zkvxc" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "kube-api-access-zkvxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:42.627963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.627832 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-envoy" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:42.627963 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.627948 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-token" pod "6bbe97ca-af94-4472-9fd2-87ea421bd627" (UID: "6bbe97ca-af94-4472-9fd2-87ea421bd627"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:42.728019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.727983 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.728019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.728011 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.728019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.728020 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkvxc\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-kube-api-access-zkvxc\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.728019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.728030 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe97ca-af94-4472-9fd2-87ea421bd627-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:42.784388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.784352 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:42.898267 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.898172 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:42.919198 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.919171 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:34:42.919867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.919841 4433 generic.go:356] "Generic (PLEG): container finished" podID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerID="79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453" exitCode=0 Oct 30 00:34:42.920004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.919884 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" Oct 30 00:34:42.920004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.919900 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" event={"ID":"6bbe97ca-af94-4472-9fd2-87ea421bd627","Type":"ContainerDied","Data":"79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453"} Oct 30 00:34:42.920004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.919949 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn" event={"ID":"6bbe97ca-af94-4472-9fd2-87ea421bd627","Type":"ContainerDied","Data":"212a99f10cb96893c11c3684345097b83b543bd10064954104064699aec871f6"} Oct 30 00:34:42.920004 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.919978 4433 scope.go:122] "RemoveContainer" containerID="79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453" Oct 30 00:34:42.931679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.931662 4433 scope.go:122] "RemoveContainer" containerID="79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453" Oct 30 00:34:42.931926 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:42.931910 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453\": container with ID starting with 79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453 not found: ID does not exist" containerID="79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453" Oct 30 00:34:42.931988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.931934 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453"} err="failed to get container status \"79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453\": rpc error: code = NotFound desc = could not find container \"79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453\": container with ID starting with 79a7ba56846903b0fbecf03ef8f66b6915928b00bdda17ae1ff6c678eb938453 not found: ID does not exist" Oct 30 00:34:42.982343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.982313 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn"] Oct 30 00:34:42.990883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:42.990851 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-fd6wn"] Oct 30 00:34:43.106607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.106548 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f"] Oct 30 00:34:43.107501 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.107468 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" containerID="cri-o://1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14" gracePeriod=30 Oct 30 00:34:43.112448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.112421 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp"] Oct 30 00:34:43.112670 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.112649 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" containerID="cri-o://f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43" gracePeriod=30 Oct 30 00:34:43.122556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.121563 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj"] Oct 30 00:34:43.132480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.127465 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" containerID="cri-o://beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f" gracePeriod=30 Oct 30 00:34:43.138787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.138755 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc"] Oct 30 00:34:43.144087 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.144037 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:43.144274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.144205 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" containerID="cri-o://e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044" gracePeriod=30 Oct 30 00:34:43.209256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.203151 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" path="/var/lib/kubelet/pods/6bbe97ca-af94-4472-9fd2-87ea421bd627/volumes" Oct 30 00:34:43.391488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.386562 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 00:34:43.391488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.386644 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 00:34:43.885403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:43.885370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:44.378083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:44.378039 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 00:34:44.378292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:44.378102 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 00:34:45.386307 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:45.386275 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:34:45.388013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:45.387985 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:34:45.436201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:45.436172 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:45.436343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:45.436223 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:45.494426 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:45.494388 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:45.815492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:45.815455 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:45.815688 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:45.815508 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:47.187154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:47.187113 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:47.784942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:47.784903 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:47.785153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:47.784957 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:48.470038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.470003 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:34:48.474668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.474640 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:34:48.548621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.544654 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:34:48.566173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566138 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-podinfo\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.566457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566203 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-data\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.566457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566255 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-socket\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.566457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566354 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-certs\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.566457 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566438 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-kube-api-access-pgj58\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-kube-api-access-pgj58\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.566799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566519 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-credential-socket\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.566799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566551 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-kube-api-access-x6nmk\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-kube-api-access-x6nmk\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.566799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566600 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-envoy\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.566799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566683 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-socket\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.566799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566713 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-envoy\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.566799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566754 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-certs\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.566799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566784 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-token\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.567135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566824 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-credential-socket\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.567135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566855 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-token\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-token\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.567135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.566937 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istiod-ca-cert\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.567135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.567008 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-podinfo\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.567135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.567044 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-data\") pod \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\" (UID: \"a7dcedc9-87ca-44ac-8ce2-2267892d50a0\") " Oct 30 00:34:48.567135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.567125 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istiod-ca-cert\") pod \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\" (UID: \"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86\") " Oct 30 00:34:48.567830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.567703 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-certs" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.567830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.567741 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-data" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.568041 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.568018 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-socket" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.568392 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.568224 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-credential-socket" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.568603 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.568502 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-credential-socket" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.568679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.568659 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.570866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.568681 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.570866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.568696 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.570866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.568725 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.570866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.569029 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istiod-ca-cert" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:48.570866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.569369 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-certs" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.571301 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.571272 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istiod-ca-cert" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:48.571786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.571752 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-data" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.572048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.572019 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-socket" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.573752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.573105 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-podinfo" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:48.573752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.573249 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-kube-api-access-x6nmk" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "kube-api-access-x6nmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.576766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.576716 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-kube-api-access-pgj58" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "kube-api-access-pgj58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.576766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.576732 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-envoy" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.576766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.576737 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-token" pod "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" (UID: "89cc8f76-f6ed-40a9-9d3a-29e03c29fd86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.578659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.578634 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-envoy" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.578790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.578663 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-token" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.578790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.578733 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-podinfo" pod "a7dcedc9-87ca-44ac-8ce2-2267892d50a0" (UID: "a7dcedc9-87ca-44ac-8ce2-2267892d50a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:48.586417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.586394 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:34:48.669339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.669245 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-envoy\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.669339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.669316 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b78a5d9-5483-4835-b906-6c13fb96461c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78a5d9-5483-4835-b906-6c13fb96461c-istiod-ca-cert\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.669917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.669355 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-certs\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.669917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.669399 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-podinfo\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.669917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.669432 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-data\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.669917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.669498 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-certs\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.670005 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b78a5d9-5483-4835-b906-6c13fb96461c-istiod-ca-cert" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.670408 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-data" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.670306 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-certs" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.669543 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-token\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-token\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.670856 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-certs" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.670805 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-kube-api-access-dcgbw\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-kube-api-access-dcgbw\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.671172 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-podinfo\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.671264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.671244 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-credential-socket\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.672255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.671348 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9acd55c2-d267-422c-9412-d6ab9621f1f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acd55c2-d267-422c-9412-d6ab9621f1f0-istiod-ca-cert\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.672255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.671375 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-credential-socket\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.673116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.671432 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-envoy\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.673208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.673154 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-kube-api-access-v47t7\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-kube-api-access-v47t7\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.673280 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.673231 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-envoy" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.673786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.673734 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9acd55c2-d267-422c-9412-d6ab9621f1f0-istiod-ca-cert" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:48.673886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.673816 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-token\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-token\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.673952 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.673938 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-socket\") pod \"9acd55c2-d267-422c-9412-d6ab9621f1f0\" (UID: \"9acd55c2-d267-422c-9412-d6ab9621f1f0\") " Oct 30 00:34:48.674037 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.674015 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-data\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.674103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.674085 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-socket\") pod \"8b78a5d9-5483-4835-b906-6c13fb96461c\" (UID: \"8b78a5d9-5483-4835-b906-6c13fb96461c\") " Oct 30 00:34:48.675091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675069 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acd55c2-d267-422c-9412-d6ab9621f1f0-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675091 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675092 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgj58\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-kube-api-access-pgj58\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675103 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675111 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6nmk\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-kube-api-access-x6nmk\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675121 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675129 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675137 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675146 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675154 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675161 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675170 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675177 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675184 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78a5d9-5483-4835-b906-6c13fb96461c-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675192 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675200 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675208 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcedc9-87ca-44ac-8ce2-2267892d50a0-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675215 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675225 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675233 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.675241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675240 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.676202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675483 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-socket" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.676202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.675638 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-credential-socket" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.678973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.678848 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-socket" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.678973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.678873 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-credential-socket" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.678973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.678855 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-data" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.678973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.678876 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-token" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.678973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.678908 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-kube-api-access-v47t7" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "kube-api-access-v47t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.679682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.679661 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-podinfo" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:48.680424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.680012 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-kube-api-access-dcgbw" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "kube-api-access-dcgbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.680424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.680066 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-podinfo" pod "8b78a5d9-5483-4835-b906-6c13fb96461c" (UID: "8b78a5d9-5483-4835-b906-6c13fb96461c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:48.680424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.680062 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-token" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.681145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.681121 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-envoy" pod "9acd55c2-d267-422c-9412-d6ab9621f1f0" (UID: "9acd55c2-d267-422c-9412-d6ab9621f1f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.687850 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.687764 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:48.776940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.776902 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcgbw\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-kube-api-access-dcgbw\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.776940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.776943 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.776959 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.776974 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.776987 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.776996 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v47t7\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-kube-api-access-v47t7\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.777005 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acd55c2-d267-422c-9412-d6ab9621f1f0-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.777014 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acd55c2-d267-422c-9412-d6ab9621f1f0-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.777022 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.777032 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78a5d9-5483-4835-b906-6c13fb96461c-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.777044 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.777148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:48.777057 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78a5d9-5483-4835-b906-6c13fb96461c-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:34:49.029083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.028992 4433 generic.go:356] "Generic (PLEG): container finished" podID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerID="f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43" exitCode=0 Oct 30 00:34:49.029252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.029184 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" Oct 30 00:34:49.030739 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.030703 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" event={"ID":"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86","Type":"ContainerDied","Data":"f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43"} Oct 30 00:34:49.030866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.030784 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp" event={"ID":"89cc8f76-f6ed-40a9-9d3a-29e03c29fd86","Type":"ContainerDied","Data":"80e23255bac36c0da34a89b275155b0d911943a6134c7e563c8fad1ce7060ca7"} Oct 30 00:34:49.030866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.030840 4433 scope.go:122] "RemoveContainer" containerID="f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43" Oct 30 00:34:49.035204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.035176 4433 generic.go:356] "Generic (PLEG): container finished" podID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerID="1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14" exitCode=0 Oct 30 00:34:49.035353 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.035333 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" Oct 30 00:34:49.035915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.035338 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" event={"ID":"a7dcedc9-87ca-44ac-8ce2-2267892d50a0","Type":"ContainerDied","Data":"1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14"} Oct 30 00:34:49.036133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.036100 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f" event={"ID":"a7dcedc9-87ca-44ac-8ce2-2267892d50a0","Type":"ContainerDied","Data":"f855f0e0812ceba8afd3b6e649bf6a464e8ccf98e5070639a0c3b59c4f333ea3"} Oct 30 00:34:49.037267 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.037246 4433 generic.go:356] "Generic (PLEG): container finished" podID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerID="beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f" exitCode=0 Oct 30 00:34:49.037499 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.037300 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" Oct 30 00:34:49.038034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.037312 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" event={"ID":"8b78a5d9-5483-4835-b906-6c13fb96461c","Type":"ContainerDied","Data":"beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f"} Oct 30 00:34:49.038253 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.038228 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj" event={"ID":"8b78a5d9-5483-4835-b906-6c13fb96461c","Type":"ContainerDied","Data":"9a6215a31020b4ae35ac24f3d16d84fd65cc9494be9fc3c2c5d996158bd8a846"} Oct 30 00:34:49.040912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.040890 4433 generic.go:356] "Generic (PLEG): container finished" podID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerID="e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044" exitCode=0 Oct 30 00:34:49.041050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.040939 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" Oct 30 00:34:49.041471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.040955 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" event={"ID":"9acd55c2-d267-422c-9412-d6ab9621f1f0","Type":"ContainerDied","Data":"e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044"} Oct 30 00:34:49.041471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.041436 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc" event={"ID":"9acd55c2-d267-422c-9412-d6ab9621f1f0","Type":"ContainerDied","Data":"2e1a504f27f08fb92e3bda1821bfa9988898c580429b66848e354fb59a66830a"} Oct 30 00:34:49.045564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.045302 4433 scope.go:122] "RemoveContainer" containerID="f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43" Oct 30 00:34:49.046376 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:49.046055 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43\": container with ID starting with f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43 not found: ID does not exist" containerID="f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43" Oct 30 00:34:49.046376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.046089 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43"} err="failed to get container status \"f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43\": rpc error: code = NotFound desc = could not find container \"f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43\": container with ID starting with f1aff0a87c2098e285c88735b15bec26be50394ac494420f34d83937bfb7cd43 not found: ID does not exist" Oct 30 00:34:49.046376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.046114 4433 scope.go:122] "RemoveContainer" containerID="1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14" Oct 30 00:34:49.058953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.058869 4433 scope.go:122] "RemoveContainer" containerID="1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14" Oct 30 00:34:49.059190 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:49.059170 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14\": container with ID starting with 1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14 not found: ID does not exist" containerID="1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14" Oct 30 00:34:49.059286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.059198 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14"} err="failed to get container status \"1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14\": rpc error: code = NotFound desc = could not find container \"1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14\": container with ID starting with 1b239b704934d041e879625dfdb0fdf0bc4997613a765c872ebd1d5782ce3a14 not found: ID does not exist" Oct 30 00:34:49.059286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.059218 4433 scope.go:122] "RemoveContainer" containerID="beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f" Oct 30 00:34:49.069857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.069838 4433 scope.go:122] "RemoveContainer" containerID="beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f" Oct 30 00:34:49.070171 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:49.070144 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f\": container with ID starting with beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f not found: ID does not exist" containerID="beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f" Oct 30 00:34:49.070287 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.070174 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f"} err="failed to get container status \"beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f\": rpc error: code = NotFound desc = could not find container \"beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f\": container with ID starting with beaa5c395d90d61c2c8bbf09a821154af9685f0cbd8e84152ec6e2a5edf36c2f not found: ID does not exist" Oct 30 00:34:49.070287 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.070194 4433 scope.go:122] "RemoveContainer" containerID="e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044" Oct 30 00:34:49.080994 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.080970 4433 scope.go:122] "RemoveContainer" containerID="e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837" Oct 30 00:34:49.092528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.092375 4433 scope.go:122] "RemoveContainer" containerID="e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044" Oct 30 00:34:49.093001 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:49.092981 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044\": container with ID starting with e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044 not found: ID does not exist" containerID="e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044" Oct 30 00:34:49.093081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.093009 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044"} err="failed to get container status \"e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044\": rpc error: code = NotFound desc = could not find container \"e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044\": container with ID starting with e7302c6b0866b288041a2582c45c39044fc25ae488026e26d7e424141b6e8044 not found: ID does not exist" Oct 30 00:34:49.093081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.093025 4433 scope.go:122] "RemoveContainer" containerID="e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837" Oct 30 00:34:49.093310 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:34:49.093294 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837\": container with ID starting with e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837 not found: ID does not exist" containerID="e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837" Oct 30 00:34:49.093364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.093315 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837"} err="failed to get container status \"e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837\": rpc error: code = NotFound desc = could not find container \"e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837\": container with ID starting with e4d490399cbd27663d664785e6c0a36d168b9863d4299df9e3fde792d1de4837 not found: ID does not exist" Oct 30 00:34:49.125974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.125947 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp"] Oct 30 00:34:49.129564 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.129539 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7pxkp"] Oct 30 00:34:49.186643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.186609 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" path="/var/lib/kubelet/pods/89cc8f76-f6ed-40a9-9d3a-29e03c29fd86/volumes" Oct 30 00:34:49.198131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.198101 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f"] Oct 30 00:34:49.202090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.202070 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-s9f7f"] Oct 30 00:34:49.234699 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.234669 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj"] Oct 30 00:34:49.239074 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.239049 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-bbjzj"] Oct 30 00:34:49.273787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.273755 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc"] Oct 30 00:34:49.279231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:49.279170 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-968jc"] Oct 30 00:34:50.244286 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:50.244248 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:51.180465 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:51.180435 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" path="/var/lib/kubelet/pods/8b78a5d9-5483-4835-b906-6c13fb96461c/volumes" Oct 30 00:34:51.180996 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:51.180979 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" path="/var/lib/kubelet/pods/9acd55c2-d267-422c-9412-d6ab9621f1f0/volumes" Oct 30 00:34:51.181481 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:51.181468 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" path="/var/lib/kubelet/pods/a7dcedc9-87ca-44ac-8ce2-2267892d50a0/volumes" Oct 30 00:34:51.434387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:51.434292 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:52.060285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.060112 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:52.170710 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.168411 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:52.944886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.944852 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k"] Oct 30 00:34:52.945484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945468 4433 cpu_manager.go:414] "containerMap: removing container" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" Oct 30 00:34:52.945526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945485 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" Oct 30 00:34:52.945526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945494 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" Oct 30 00:34:52.945526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945499 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" Oct 30 00:34:52.945526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945508 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" containerName="maas-api" Oct 30 00:34:52.945526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945515 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" containerName="maas-api" Oct 30 00:34:52.945526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945525 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945530 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945538 4433 cpu_manager.go:414] "containerMap: removing container" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945543 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945562 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945566 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945590 4433 cpu_manager.go:414] "containerMap: removing container" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945595 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945610 4433 cpu_manager.go:414] "containerMap: removing container" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945615 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945626 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945630 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945647 4433 cpu_manager.go:414] "containerMap: removing container" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945652 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945668 4433 cpu_manager.go:414] "containerMap: removing container" podUID="6f1ef643-a2f0-4cd2-8788-08b8084b0dca" containerName="maas-api" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945672 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1ef643-a2f0-4cd2-8788-08b8084b0dca" containerName="maas-api" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945678 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" Oct 30 00:34:52.945755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945683 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945782 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1ef643-a2f0-4cd2-8788-08b8084b0dca" containerName="maas-api" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945789 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f310099f-ae88-4cfe-9ee1-3175b696ab12" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945797 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b78a5d9-5483-4835-b906-6c13fb96461c" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945803 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dcedc9-87ca-44ac-8ce2-2267892d50a0" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945809 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" containerName="maas-api" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945814 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2186b19c-57bf-41cb-bf4e-a9800a8e2934" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945819 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4612bf-5599-477a-9abe-c2a15751d372" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945827 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945833 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b75034-6711-4f76-9105-32372500328c" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945841 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cc8f76-f6ed-40a9-9d3a-29e03c29fd86" containerName="istio-proxy" Oct 30 00:34:52.946230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.945850 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbe97ca-af94-4472-9fd2-87ea421bd627" containerName="istio-proxy" Oct 30 00:34:52.951333 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.951314 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:52.969444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.969414 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k"] Oct 30 00:34:52.978798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.978763 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c1eeb6\"/\"default-dockercfg-jmf2t\"" Oct 30 00:34:52.978948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.978895 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c1eeb6\"/\"auth-test-model-c1eeb6-kserve-self-signed-certs\"" Oct 30 00:34:52.981634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.981553 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c1eeb6\"/\"kube-root-ca.crt\"" Oct 30 00:34:52.981765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.981664 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c1eeb6\"/\"openshift-service-ca.crt\"" Oct 30 00:34:52.983226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:52.983201 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:53.052361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.052319 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-home\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.052539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.052480 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tmp-dir\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.052539 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.052521 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-model-cache\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.052693 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.052667 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-dshm\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.052784 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.052704 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tls-certs\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.052839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.052830 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg26z\" (UniqueName: \"kubernetes.io/projected/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-kube-api-access-sg26z\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.156528 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg26z\" (UniqueName: \"kubernetes.io/projected/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-kube-api-access-sg26z\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.156670 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-home\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.156785 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tmp-dir\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.156824 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-model-cache\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.156904 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-dshm\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.156934 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tls-certs\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.159151 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-home\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.159454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.159449 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tmp-dir\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.171743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.161969 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-model-cache\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.190416 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.190381 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-dshm\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.205718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.204185 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg26z\" (UniqueName: \"kubernetes.io/projected/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-kube-api-access-sg26z\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.213757 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.213559 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tls-certs\") pod \"auth-test-model-c1eeb6-kserve-58545675cc-rf49k\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.264453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.264400 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:53.589866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.589836 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k"] Oct 30 00:34:53.590442 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:34:53.590419 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7ba6c4_0c1c_4515_82f1_de9f90cc6a77.slice/crio-803c678f1eccaad8a8b27a9da493eada42ce460853868e63ec6a2a147c8e069d WatchSource:0}: Error finding container 803c678f1eccaad8a8b27a9da493eada42ce460853868e63ec6a2a147c8e069d: Status 404 returned error can't find the container with id 803c678f1eccaad8a8b27a9da493eada42ce460853868e63ec6a2a147c8e069d Oct 30 00:34:53.829588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:53.829532 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:54.114065 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:54.114030 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" event={"ID":"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77","Type":"ContainerStarted","Data":"23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c"} Oct 30 00:34:54.114494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:54.114086 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" event={"ID":"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77","Type":"ContainerStarted","Data":"803c678f1eccaad8a8b27a9da493eada42ce460853868e63ec6a2a147c8e069d"} Oct 30 00:34:54.158760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:54.158709 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" podStartSLOduration=2.158695847 podStartE2EDuration="2.158695847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:54.156906759 +0000 UTC m=+1803.090975294" watchObservedRunningTime="2026-10-30 00:34:54.158695847 +0000 UTC m=+1803.092764381" Oct 30 00:34:54.271964 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:54.271855 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:34:55.214939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.214889 4433 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","poda98846a5-27d4-4a29-82ce-a573f0a1a3eb"] err="unable to destroy cgroup paths for cgroup [kubepods burstable poda98846a5-27d4-4a29-82ce-a573f0a1a3eb] : Timed out while waiting for systemd to remove kubepods-burstable-poda98846a5_27d4_4a29_82ce_a573f0a1a3eb.slice" Oct 30 00:34:55.214939 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.214938 4433 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="a98846a5-27d4-4a29-82ce-a573f0a1a3eb" err="failed to delete cgroup paths for [kubepods burstable poda98846a5-27d4-4a29-82ce-a573f0a1a3eb] : unable to destroy cgroup paths for cgroup [kubepods burstable poda98846a5-27d4-4a29-82ce-a573f0a1a3eb] : Timed out while waiting for systemd to remove kubepods-burstable-poda98846a5_27d4_4a29_82ce_a573f0a1a3eb.slice" Oct 30 00:34:55.367135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.367097 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:55.481152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.481063 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:34:55.481325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.481282 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:34:55.567226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.567193 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh"] Oct 30 00:34:55.567992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.567974 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acd55c2-d267-422c-9412-d6ab9621f1f0" containerName="istio-proxy" Oct 30 00:34:55.573666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.573642 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.592967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.592937 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh"] Oct 30 00:34:55.601623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.601375 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-29f3ca\"/\"openshift-service-ca.crt\"" Oct 30 00:34:55.602062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.602032 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-29f3ca\"/\"auth-test-model-29f3ca-kserve-self-signed-certs\"" Oct 30 00:34:55.603086 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.603067 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-29f3ca\"/\"kube-root-ca.crt\"" Oct 30 00:34:55.603185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.603080 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-29f3ca\"/\"default-dockercfg-6vs6b\"" Oct 30 00:34:55.619687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.619651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:55.710986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.710945 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tls-certs\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.711170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.711023 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-home\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.711170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.711091 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xnz\" (UniqueName: \"kubernetes.io/projected/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-kube-api-access-b4xnz\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.711170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.711164 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-dshm\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.711314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.711302 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tmp-dir\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.711370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.711359 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-model-cache\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.730492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.730454 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:55.817595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.817253 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-home\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.817595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.817305 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xnz\" (UniqueName: \"kubernetes.io/projected/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-kube-api-access-b4xnz\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.817595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.817369 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-dshm\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.817595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.817586 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tmp-dir\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.817974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.817679 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-model-cache\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.817974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.817732 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tls-certs\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.827613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.818299 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-home\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.827613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.818894 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tmp-dir\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.827613 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.818943 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-model-cache\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.832335 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.831776 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-dshm\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.836034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.834251 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tls-certs\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.848050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.847372 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xnz\" (UniqueName: \"kubernetes.io/projected/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-kube-api-access-b4xnz\") pod \"auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:55.890250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:55.890217 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:56.212314 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:56.209110 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh"] Oct 30 00:34:56.212986 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:34:56.212810 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f92ed7a_b83b_47e0_8778_573e8f1d6efc.slice/crio-ca5c6b17664aedc335e0cb372408ef9c3a2165e16c87e987188044010d997322 WatchSource:0}: Error finding container ca5c6b17664aedc335e0cb372408ef9c3a2165e16c87e987188044010d997322: Status 404 returned error can't find the container with id ca5c6b17664aedc335e0cb372408ef9c3a2165e16c87e987188044010d997322 Oct 30 00:34:56.940416 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:56.940380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:57.136450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:57.136413 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" event={"ID":"2f92ed7a-b83b-47e0-8778-573e8f1d6efc","Type":"ContainerStarted","Data":"32c581951afa1fc5878a3f9b9c74d5a8d3bdc02781ebe7eda15917ed6769a7e5"} Oct 30 00:34:57.136450 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:57.136453 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" event={"ID":"2f92ed7a-b83b-47e0-8778-573e8f1d6efc","Type":"ContainerStarted","Data":"ca5c6b17664aedc335e0cb372408ef9c3a2165e16c87e987188044010d997322"} Oct 30 00:34:57.180679 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:57.180627 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" podStartSLOduration=2.180608774 podStartE2EDuration="2.180608774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:57.17905803 +0000 UTC m=+1806.113126566" watchObservedRunningTime="2026-10-30 00:34:57.180608774 +0000 UTC m=+1806.114677309" Oct 30 00:34:57.891059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:57.891019 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:34:57.955915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:57.955861 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:58.467453 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:58.467412 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:59.990550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:34:59.990514 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:01.488929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:01.488878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:02.297186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:02.297144 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:02.404773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:02.404732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:03.179177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:03.179133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:03.266061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:03.266030 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:35:03.282367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:03.282327 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:35:04.231376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:04.231348 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:35:04.903221 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:04.903175 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:05.891266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:05.891229 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:35:05.903931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:05.903905 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:35:06.254110 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:06.254025 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:35:06.428192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:06.428160 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:07.102066 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:35:07.102030 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:35:07.859390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:07.859361 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:09.387553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:09.387518 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:10.894096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:10.894059 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:12.383463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:12.383427 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:13.897879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:13.897842 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:15.564019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:15.563981 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:17.023073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:17.023040 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:18.498622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:18.498562 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:20.089352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:20.089308 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:21.541634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:21.541567 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:21.651266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:21.651226 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:22.896192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:22.896155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:23.337161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:23.337124 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:23.456464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:23.456425 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:23.579527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:23.579474 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:23.978668 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:23.978623 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:25.112277 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:25.112238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:26.681957 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:26.681916 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:28.166261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:28.166224 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:29.771447 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.771409 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4"] Oct 30 00:35:29.777741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.777709 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.803928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.803872 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:29.810385 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.810341 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4"] Oct 30 00:35:29.820321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.819915 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-ecb34a\"/\"openshift-service-ca.crt\"" Oct 30 00:35:29.820321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.820028 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-ecb34a\"/\"auth-test-model-ecb34a-kserve-self-signed-certs\"" Oct 30 00:35:29.820321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.820060 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-ecb34a\"/\"kube-root-ca.crt\"" Oct 30 00:35:29.820321 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.820213 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-ecb34a\"/\"default-dockercfg-4jt7r\"" Oct 30 00:35:29.889737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.889687 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-tmp-dir\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.889966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.889828 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-model-cache\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.889966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.889885 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a999892-3977-46db-89f3-1cf2f172a2e5-tls-certs\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.890116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.889964 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-dshm\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.890116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.890065 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-home\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.890226 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.890210 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2nd\" (UniqueName: \"kubernetes.io/projected/4a999892-3977-46db-89f3-1cf2f172a2e5-kube-api-access-hd2nd\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.992556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.992507 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2nd\" (UniqueName: \"kubernetes.io/projected/4a999892-3977-46db-89f3-1cf2f172a2e5-kube-api-access-hd2nd\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.992961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.992931 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-tmp-dir\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.993144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.993119 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-model-cache\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.994079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.993745 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-model-cache\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.994246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.993959 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-tmp-dir\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.994553 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.994524 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a999892-3977-46db-89f3-1cf2f172a2e5-tls-certs\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.994735 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.994642 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-dshm\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.994811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.994745 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-home\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.995139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.995118 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-home\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.996972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.996947 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-dshm\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.997055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.997028 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a999892-3977-46db-89f3-1cf2f172a2e5-tls-certs\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:29.998826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:29.998788 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:30.021156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:30.021126 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2nd\" (UniqueName: \"kubernetes.io/projected/4a999892-3977-46db-89f3-1cf2f172a2e5-kube-api-access-hd2nd\") pod \"auth-test-model-ecb34a-kserve-f548b5745-dfdq4\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:30.121244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:30.121205 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:30.518271 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:35:30.518241 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a999892_3977_46db_89f3_1cf2f172a2e5.slice/crio-a4083f6346b89e76fd7eec3c622dedcb36df57b9191c4fd876045595b375a106 WatchSource:0}: Error finding container a4083f6346b89e76fd7eec3c622dedcb36df57b9191c4fd876045595b375a106: Status 404 returned error can't find the container with id a4083f6346b89e76fd7eec3c622dedcb36df57b9191c4fd876045595b375a106 Oct 30 00:35:30.518631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:30.518606 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4"] Oct 30 00:35:31.081815 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:31.081781 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:31.203826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:31.203792 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:31.410867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:31.410774 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:31.429343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:31.429305 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" event={"ID":"4a999892-3977-46db-89f3-1cf2f172a2e5","Type":"ContainerStarted","Data":"592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade"} Oct 30 00:35:31.429540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:31.429355 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" event={"ID":"4a999892-3977-46db-89f3-1cf2f172a2e5","Type":"ContainerStarted","Data":"a4083f6346b89e76fd7eec3c622dedcb36df57b9191c4fd876045595b375a106"} Oct 30 00:35:31.472627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:31.472561 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" podStartSLOduration=2.472542385 podStartE2EDuration="2.472542385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:31.4716667 +0000 UTC m=+1840.405735272" watchObservedRunningTime="2026-10-30 00:35:31.472542385 +0000 UTC m=+1840.406610934" Oct 30 00:35:31.521732 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:31.521697 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:32.098865 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.098826 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:32.122910 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.122875 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:32.603018 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.602980 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq"] Oct 30 00:35:32.608484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.608458 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638010 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.637949 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq"] Oct 30 00:35:32.638255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638054 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-socket\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638386 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638329 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-token\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638404 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzxn\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-kube-api-access-kgzxn\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638535 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-credential-socket\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638754 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638729 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-data\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638782 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-podinfo\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638807 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-envoy\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638919 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istiod-ca-cert\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.638992 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.638971 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-certs\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.640284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.640257 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-1dc29b-openshift-default-dockercfg-c6qps\"" Oct 30 00:35:32.665108 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.664117 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.745624 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-socket\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.745735 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-token\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.745763 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzxn\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-kube-api-access-kgzxn\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.745845 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-credential-socket\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.745893 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-data\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.745936 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-podinfo\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.745965 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-envoy\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.746091 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istiod-ca-cert\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.746138 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-certs\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.746666 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-certs\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.746919 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-data\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.747462 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-socket\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.748091 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istiod-ca-cert\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.759597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.748787 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-credential-socket\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.784592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.774865 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-envoy\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.784592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.774995 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-podinfo\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.792595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.791352 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-token\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.801598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.793496 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzxn\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-kube-api-access-kgzxn\") pod \"e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.926766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.926653 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:32.996627 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:32.996555 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:33.107276 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.107216 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:33.267800 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.267771 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq"] Oct 30 00:35:33.268342 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:35:33.268315 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3c1949_7d0e_41db_ab76_fa622a2c29f7.slice/crio-0c94b24c6b5b22066c336cc768c3c0e6873245d59d3c26ec252cffbfc2a3c87b WatchSource:0}: Error finding container 0c94b24c6b5b22066c336cc768c3c0e6873245d59d3c26ec252cffbfc2a3c87b: Status 404 returned error can't find the container with id 0c94b24c6b5b22066c336cc768c3c0e6873245d59d3c26ec252cffbfc2a3c87b Oct 30 00:35:33.271246 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.271211 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:35:33.271327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.271279 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:35:33.271327 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.271315 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:35:33.442887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.442851 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" event={"ID":"7f3c1949-7d0e-41db-ab76-fa622a2c29f7","Type":"ContainerStarted","Data":"3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc"} Oct 30 00:35:33.443096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.442905 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" event={"ID":"7f3c1949-7d0e-41db-ab76-fa622a2c29f7","Type":"ContainerStarted","Data":"0c94b24c6b5b22066c336cc768c3c0e6873245d59d3c26ec252cffbfc2a3c87b"} Oct 30 00:35:33.490966 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.490907 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" podStartSLOduration=1.490893651 podStartE2EDuration="1.490893651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:33.488552828 +0000 UTC m=+1842.422621404" watchObservedRunningTime="2026-10-30 00:35:33.490893651 +0000 UTC m=+1842.424962237" Oct 30 00:35:33.858210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.858158 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:33.927224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.927189 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:33.971542 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:33.971507 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:34.164359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:34.164258 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:34.934612 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:34.934498 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:35.012720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:35.012682 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:35.455932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:35.455900 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:35.457084 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:35.457057 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:35:36.381595 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:36.381534 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:37.101434 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:35:37.101397 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:35:37.906680 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:37.906644 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:39.337493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:39.337462 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:39.920634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:39.920590 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:40.123319 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:40.123282 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:40.136188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:40.136161 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:40.503463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:40.503431 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:35:40.532397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:40.532371 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:40.768735 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:40.768646 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:42.382034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:42.381976 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:43.876364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:43.876334 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:45.486916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:45.486878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:47.176153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:47.176114 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:48.619532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:48.619498 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:50.136248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:50.136207 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:51.649111 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:51.649068 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:53.199762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:53.199725 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:54.176676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:54.176637 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:54.786859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:54.786826 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:56.384360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:56.384322 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:56.492399 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:56.492358 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:58.004518 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:58.004482 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:59.383291 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.383252 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct"] Oct 30 00:35:59.388231 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.388203 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.418224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.418196 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1dc29b\"/\"openshift-service-ca.crt\"" Oct 30 00:35:59.418432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.418403 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1dc29b\"/\"noten-model-1dc29b-kserve-self-signed-certs\"" Oct 30 00:35:59.418432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.418229 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1dc29b\"/\"default-dockercfg-bzn6h\"" Oct 30 00:35:59.418677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.418229 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1dc29b\"/\"kube-root-ca.crt\"" Oct 30 00:35:59.422349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.422326 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct"] Oct 30 00:35:59.446883 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.446851 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:59.487756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.487727 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tls-certs\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.487756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.487758 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trhwf\" (UniqueName: \"kubernetes.io/projected/4f25eff0-d15b-4953-ac08-65f5b86b15bf-kube-api-access-trhwf\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.488048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.488022 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tmp-dir\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.488123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.488098 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-model-cache\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.488230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.488214 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-dshm\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.488394 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.488372 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-home\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.550969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.550935 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:59.589396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.589362 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-home\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.589625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.589441 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tls-certs\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.589625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.589466 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trhwf\" (UniqueName: \"kubernetes.io/projected/4f25eff0-d15b-4953-ac08-65f5b86b15bf-kube-api-access-trhwf\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.589775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.589635 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tmp-dir\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.589775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.589685 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-model-cache\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.589968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.589777 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-dshm\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.589968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.589828 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-home\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.590082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.590012 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tmp-dir\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.590141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.590115 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-model-cache\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.592028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.592005 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-dshm\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.592214 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.592195 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tls-certs\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.670377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.670276 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trhwf\" (UniqueName: \"kubernetes.io/projected/4f25eff0-d15b-4953-ac08-65f5b86b15bf-kube-api-access-trhwf\") pod \"noten-model-1dc29b-kserve-7f6cfd86df-jnzct\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:35:59.702886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:35:59.702848 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:36:00.033088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:00.033057 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct"] Oct 30 00:36:00.033747 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:36:00.033721 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f25eff0_d15b_4953_ac08_65f5b86b15bf.slice/crio-5f858abc5e51de3b1c67055b50e66e15e1b8b2ebeaf014a081c9a008897a7306 WatchSource:0}: Error finding container 5f858abc5e51de3b1c67055b50e66e15e1b8b2ebeaf014a081c9a008897a7306: Status 404 returned error can't find the container with id 5f858abc5e51de3b1c67055b50e66e15e1b8b2ebeaf014a081c9a008897a7306 Oct 30 00:36:00.709415 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:00.709370 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" event={"ID":"4f25eff0-d15b-4953-ac08-65f5b86b15bf","Type":"ContainerStarted","Data":"2d636e126add37abf1eeccf4d091cfc62f39f52675df3ba12c233171531c2d88"} Oct 30 00:36:00.709832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:00.709427 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" event={"ID":"4f25eff0-d15b-4953-ac08-65f5b86b15bf","Type":"ContainerStarted","Data":"5f858abc5e51de3b1c67055b50e66e15e1b8b2ebeaf014a081c9a008897a7306"} Oct 30 00:36:00.752788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:00.752735 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" podStartSLOduration=1.7527201909999999 podStartE2EDuration="1.752720191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:00.752359332 +0000 UTC m=+1869.686427882" watchObservedRunningTime="2026-10-30 00:36:00.752720191 +0000 UTC m=+1869.686788726" Oct 30 00:36:01.095628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:01.095590 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:01.703418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:01.703351 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:36:02.573761 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:02.573721 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:04.062245 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:04.062207 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:05.504202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:05.504170 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:07.024937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:07.024903 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:07.123345 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:36:07.123301 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:36:08.895728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:08.895689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:09.703225 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:09.703188 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:36:09.716064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:09.716039 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:36:09.781584 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:09.781542 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:36:10.192694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:10.192654 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:11.797924 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:11.797875 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:13.402974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:13.402936 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:14.880835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:14.880800 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:16.245343 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:16.245304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:16.482158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:16.482122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:18.123799 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:18.123757 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:19.547631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:19.547597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:21.121081 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:21.121027 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:22.596642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:22.596603 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:24.168968 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:24.168925 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:24.686947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:24.686906 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct"] Oct 30 00:36:24.725349 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:24.725319 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:24.840970 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:24.839854 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq"] Oct 30 00:36:24.840970 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:24.840114 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" podUID="7f3c1949-7d0e-41db-ab76-fa622a2c29f7" containerName="istio-proxy" containerID="cri-o://3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc" gracePeriod=30 Oct 30 00:36:24.898631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:24.898389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:25.304653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:25.304445 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:25.412292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:25.412255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:25.688102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:25.688069 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:27.211758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:27.211708 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:28.785873 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:28.785838 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:30.127009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.126950 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:36:30.211540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.211509 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:30.303681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.303641 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-data\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.303892 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.303794 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-token\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-token\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.303892 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.303869 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-socket\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.304014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.303895 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-data" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:30.304014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.303905 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istiod-ca-cert\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.304120 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.304065 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-credential-socket\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.304223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.304202 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-certs\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.304287 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istiod-ca-cert" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.304329 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-envoy\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.304377 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-podinfo\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.304439 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-kube-api-access-kgzxn\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-kube-api-access-kgzxn\") pod \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\" (UID: \"7f3c1949-7d0e-41db-ab76-fa622a2c29f7\") " Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.304952 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-certs" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.305313 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.305337 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.305350 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.308904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.305872 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-credential-socket" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:30.309653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.309378 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-kube-api-access-kgzxn" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "kube-api-access-kgzxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:36:30.309653 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.309435 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-envoy" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:30.309775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.309717 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-token" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:36:30.309915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.309885 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-socket" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:30.315033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.314998 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-podinfo" pod "7f3c1949-7d0e-41db-ab76-fa622a2c29f7" (UID: "7f3c1949-7d0e-41db-ab76-fa622a2c29f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:36:30.407424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.407346 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.407424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.407377 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.407424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.407386 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.407424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.407394 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzxn\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-kube-api-access-kgzxn\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.407424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.407403 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.407424 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.407411 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3c1949-7d0e-41db-ab76-fa622a2c29f7-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:30.430135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.430101 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:30.944830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.944790 4433 generic.go:356] "Generic (PLEG): container finished" podID="7f3c1949-7d0e-41db-ab76-fa622a2c29f7" containerID="3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc" exitCode=0 Oct 30 00:36:30.945028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.944838 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" Oct 30 00:36:30.945028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.944861 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" event={"ID":"7f3c1949-7d0e-41db-ab76-fa622a2c29f7","Type":"ContainerDied","Data":"3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc"} Oct 30 00:36:30.945028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.944911 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq" event={"ID":"7f3c1949-7d0e-41db-ab76-fa622a2c29f7","Type":"ContainerDied","Data":"0c94b24c6b5b22066c336cc768c3c0e6873245d59d3c26ec252cffbfc2a3c87b"} Oct 30 00:36:30.945028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.944940 4433 scope.go:122] "RemoveContainer" containerID="3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc" Oct 30 00:36:30.957812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.957793 4433 scope.go:122] "RemoveContainer" containerID="3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc" Oct 30 00:36:30.958081 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:36:30.958062 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc\": container with ID starting with 3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc not found: ID does not exist" containerID="3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc" Oct 30 00:36:30.958134 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:30.958088 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc"} err="failed to get container status \"3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc\": rpc error: code = NotFound desc = could not find container \"3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc\": container with ID starting with 3f2c9157d50e9db64ed21acd5849f9366384418a28f2986bcc99cdad0287bafc not found: ID does not exist" Oct 30 00:36:31.014626 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:31.014596 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq"] Oct 30 00:36:31.020075 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:31.020045 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-1dc29b-openshift-default-756b9c95d7-lpzdq"] Oct 30 00:36:31.182169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:31.182134 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3c1949-7d0e-41db-ab76-fa622a2c29f7" path="/var/lib/kubelet/pods/7f3c1949-7d0e-41db-ab76-fa622a2c29f7/volumes" Oct 30 00:36:31.646405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:31.646366 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:33.065117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:33.065082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:34.582376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:34.582345 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:36.070829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:36.070794 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:37.100759 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:36:37.100724 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:36:37.595214 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:37.595164 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:39.165673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:39.165635 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:39.712179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:39.712135 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" podUID="4f25eff0-d15b-4953-ac08-65f5b86b15bf" containerName="main" containerID="cri-o://2d636e126add37abf1eeccf4d091cfc62f39f52675df3ba12c233171531c2d88" gracePeriod=45 Oct 30 00:36:39.771482 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:39.771444 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" podUID="4f25eff0-d15b-4953-ac08-65f5b86b15bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": dial tcp 10.134.0.147:8000: connect: connection refused" Oct 30 00:36:40.011288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.011254 4433 generic.go:356] "Generic (PLEG): container finished" podID="4f25eff0-d15b-4953-ac08-65f5b86b15bf" containerID="2d636e126add37abf1eeccf4d091cfc62f39f52675df3ba12c233171531c2d88" exitCode=0 Oct 30 00:36:40.011462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.011341 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" event={"ID":"4f25eff0-d15b-4953-ac08-65f5b86b15bf","Type":"ContainerDied","Data":"2d636e126add37abf1eeccf4d091cfc62f39f52675df3ba12c233171531c2d88"} Oct 30 00:36:40.011462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.011393 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" event={"ID":"4f25eff0-d15b-4953-ac08-65f5b86b15bf","Type":"ContainerDied","Data":"5f858abc5e51de3b1c67055b50e66e15e1b8b2ebeaf014a081c9a008897a7306"} Oct 30 00:36:40.011462 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.011413 4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f858abc5e51de3b1c67055b50e66e15e1b8b2ebeaf014a081c9a008897a7306" Oct 30 00:36:40.017115 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.017092 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:36:40.099337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.099295 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:40.146433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146398 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tmp-dir\") pod \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " Oct 30 00:36:40.146631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146458 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-home\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-home\") pod \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " Oct 30 00:36:40.146631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146518 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tls-certs\") pod \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " Oct 30 00:36:40.146741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146642 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-model-cache\") pod \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " Oct 30 00:36:40.146741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146694 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-dshm\") pod \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " Oct 30 00:36:40.146741 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146711 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tmp-dir" pod "4f25eff0-d15b-4953-ac08-65f5b86b15bf" (UID: "4f25eff0-d15b-4953-ac08-65f5b86b15bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:40.146912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146793 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-home" pod "4f25eff0-d15b-4953-ac08-65f5b86b15bf" (UID: "4f25eff0-d15b-4953-ac08-65f5b86b15bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:40.146912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146845 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f25eff0-d15b-4953-ac08-65f5b86b15bf-kube-api-access-trhwf\" (UniqueName: \"kubernetes.io/projected/4f25eff0-d15b-4953-ac08-65f5b86b15bf-kube-api-access-trhwf\") pod \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\" (UID: \"4f25eff0-d15b-4953-ac08-65f5b86b15bf\") " Oct 30 00:36:40.147005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.146918 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-model-cache" pod "4f25eff0-d15b-4953-ac08-65f5b86b15bf" (UID: "4f25eff0-d15b-4953-ac08-65f5b86b15bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:40.147431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.147408 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:40.147431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.147430 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:40.147634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.147439 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:40.149046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.149025 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-dshm" pod "4f25eff0-d15b-4953-ac08-65f5b86b15bf" (UID: "4f25eff0-d15b-4953-ac08-65f5b86b15bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:40.149126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.149044 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tls-certs" pod "4f25eff0-d15b-4953-ac08-65f5b86b15bf" (UID: "4f25eff0-d15b-4953-ac08-65f5b86b15bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:36:40.149126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.149064 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f25eff0-d15b-4953-ac08-65f5b86b15bf-kube-api-access-trhwf" pod "4f25eff0-d15b-4953-ac08-65f5b86b15bf" (UID: "4f25eff0-d15b-4953-ac08-65f5b86b15bf"). InnerVolumeSpecName "kube-api-access-trhwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:36:40.248708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.248624 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25eff0-d15b-4953-ac08-65f5b86b15bf-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:40.248708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.248653 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25eff0-d15b-4953-ac08-65f5b86b15bf-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:40.248708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.248665 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trhwf\" (UniqueName: \"kubernetes.io/projected/4f25eff0-d15b-4953-ac08-65f5b86b15bf-kube-api-access-trhwf\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:36:40.261778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.261746 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:40.614934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:40.614889 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:41.018352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:41.018259 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct" Oct 30 00:36:41.073400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:41.073364 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct"] Oct 30 00:36:41.087917 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:41.087883 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-1dc29b/noten-model-1dc29b-kserve-7f6cfd86df-jnzct"] Oct 30 00:36:41.181698 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:41.181666 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f25eff0-d15b-4953-ac08-65f5b86b15bf" path="/var/lib/kubelet/pods/4f25eff0-d15b-4953-ac08-65f5b86b15bf/volumes" Oct 30 00:36:42.022857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:42.022825 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:43.827148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:43.827111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:45.411614 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:45.411550 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:46.992475 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:46.992436 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:47.422288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:47.422250 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:47.523230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:47.523198 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:48.574786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:48.574748 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:50.086713 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:50.086674 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:51.670427 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:51.670385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:53.182777 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:53.182746 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:54.577890 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:54.577861 4433 scope.go:122] "RemoveContainer" containerID="055c0a9252aa64232e83a07b83dfbf0bfa1d76ebfb07d9182c213135e2e3f0e4" Oct 30 00:36:54.601758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:54.601719 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:56.027543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:56.027485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:57.566532 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:57.566492 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:59.009478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:36:59.009440 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:00.465145 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:00.465109 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:01.993619 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:01.993555 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:03.589643 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:03.589566 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:05.259154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:05.259064 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:06.753236 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:06.753198 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:07.099431 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:07.099395 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:37:07.408014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:07.407926 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:07.569238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:07.569202 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:08.427485 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:08.427441 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k"] Oct 30 00:37:08.465103 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:08.465067 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:08.612919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:08.612877 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:09.207604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:09.207538 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:09.310497 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:09.310464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:10.235148 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:10.235105 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:11.681694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:11.681660 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:12.553912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:12.553873 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr"] Oct 30 00:37:12.638347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:12.638311 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:13.270026 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:13.269985 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:14.708422 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:14.708389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:16.222371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:16.222331 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:17.800737 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:17.800697 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:18.412798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:18.412748 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:18.525797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:18.523016 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:19.397766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:19.397724 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:20.979676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:20.979643 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:22.463527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.463493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:22.929047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.929014 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s"] Oct 30 00:37:22.929692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.929674 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7f3c1949-7d0e-41db-ab76-fa622a2c29f7" containerName="istio-proxy" Oct 30 00:37:22.929692 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.929690 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3c1949-7d0e-41db-ab76-fa622a2c29f7" containerName="istio-proxy" Oct 30 00:37:22.929853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.929717 4433 cpu_manager.go:414] "containerMap: removing container" podUID="4f25eff0-d15b-4953-ac08-65f5b86b15bf" containerName="main" Oct 30 00:37:22.929853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.929722 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f25eff0-d15b-4953-ac08-65f5b86b15bf" containerName="main" Oct 30 00:37:22.929853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.929834 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3c1949-7d0e-41db-ab76-fa622a2c29f7" containerName="istio-proxy" Oct 30 00:37:22.929853 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.929848 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f25eff0-d15b-4953-ac08-65f5b86b15bf" containerName="main" Oct 30 00:37:22.933600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.933566 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:22.949383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.949356 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-63e288-openshift-default-dockercfg-d6zj8\"" Oct 30 00:37:22.968455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:22.968414 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s"] Oct 30 00:37:23.022746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.022711 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-credential-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.022915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.022776 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-certs\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.022915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.022828 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.022915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.022862 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26be440e-117f-46bd-957c-f59d9616abcb-istiod-ca-cert\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.023038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.022961 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-istio-token\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.023038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.023012 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-data\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.023171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.023157 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26be440e-117f-46bd-957c-f59d9616abcb-istio-podinfo\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.023207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.023187 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfdz\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-kube-api-access-mtfdz\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.023300 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.023288 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-envoy\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125172 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26be440e-117f-46bd-957c-f59d9616abcb-istio-podinfo\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125218 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfdz\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-kube-api-access-mtfdz\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125297 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-envoy\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125354 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-credential-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125408 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-certs\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125432 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125474 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26be440e-117f-46bd-957c-f59d9616abcb-istiod-ca-cert\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125542 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-istio-token\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125605 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-data\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.125863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125835 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-credential-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.126005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.125984 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.126101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.126075 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-certs\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.126157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.126121 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-data\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.126241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.126223 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26be440e-117f-46bd-957c-f59d9616abcb-istiod-ca-cert\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.127791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.127769 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-envoy\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.127969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.127951 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26be440e-117f-46bd-957c-f59d9616abcb-istio-podinfo\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.149500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.149463 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-istio-token\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.149649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.149597 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfdz\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-kube-api-access-mtfdz\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.247781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.247687 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:23.452899 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.452838 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" podUID="4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" containerName="main" containerID="cri-o://23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c" gracePeriod=45 Oct 30 00:37:23.573778 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.573748 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s"] Oct 30 00:37:23.574948 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:37:23.574914 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26be440e_117f_46bd_957c_f59d9616abcb.slice/crio-606d1cf18dcb3b8803e83958162456b97b5df7aab4eff9219665b0a2ac07adfa WatchSource:0}: Error finding container 606d1cf18dcb3b8803e83958162456b97b5df7aab4eff9219665b0a2ac07adfa: Status 404 returned error can't find the container with id 606d1cf18dcb3b8803e83958162456b97b5df7aab4eff9219665b0a2ac07adfa Oct 30 00:37:23.578302 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.578262 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:37:23.578400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.578382 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:37:23.578445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.578430 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:37:23.733375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.733345 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:37:23.838746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.838646 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-kube-api-access-sg26z\" (UniqueName: \"kubernetes.io/projected/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-kube-api-access-sg26z\") pod \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " Oct 30 00:37:23.838931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.838809 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-dshm\") pod \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " Oct 30 00:37:23.838931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.838838 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tls-certs\") pod \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " Oct 30 00:37:23.838931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.838903 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tmp-dir\") pod \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " Oct 30 00:37:23.839116 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.838965 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-model-cache\") pod \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " Oct 30 00:37:23.839181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.839119 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-home\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-home\") pod \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\" (UID: \"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77\") " Oct 30 00:37:23.839269 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.839244 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tmp-dir" pod "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" (UID: "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:23.839350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.839316 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-model-cache" pod "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" (UID: "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:23.839536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.839516 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-home" pod "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" (UID: "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:23.839764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.839747 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:23.839845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.839765 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:23.839845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.839774 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:23.841133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.841112 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-kube-api-access-sg26z" pod "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" (UID: "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77"). InnerVolumeSpecName "kube-api-access-sg26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:23.841371 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.841356 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-dshm" pod "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" (UID: "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:23.841431 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.841373 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tls-certs" pod "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" (UID: "4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:23.941176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.941139 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg26z\" (UniqueName: \"kubernetes.io/projected/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-kube-api-access-sg26z\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:23.941176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.941174 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:23.941176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:23.941184 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:24.278206 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.278174 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:24.389519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.389472 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" event={"ID":"26be440e-117f-46bd-957c-f59d9616abcb","Type":"ContainerStarted","Data":"0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52"} Oct 30 00:37:24.389801 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.389549 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" event={"ID":"26be440e-117f-46bd-957c-f59d9616abcb","Type":"ContainerStarted","Data":"606d1cf18dcb3b8803e83958162456b97b5df7aab4eff9219665b0a2ac07adfa"} Oct 30 00:37:24.396412 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.395565 4433 generic.go:356] "Generic (PLEG): container finished" podID="4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" containerID="23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c" exitCode=0 Oct 30 00:37:24.396412 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.395719 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" event={"ID":"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77","Type":"ContainerDied","Data":"23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c"} Oct 30 00:37:24.396412 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.395752 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" Oct 30 00:37:24.396412 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.395755 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k" event={"ID":"4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77","Type":"ContainerDied","Data":"803c678f1eccaad8a8b27a9da493eada42ce460853868e63ec6a2a147c8e069d"} Oct 30 00:37:24.396412 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.395778 4433 scope.go:122] "RemoveContainer" containerID="23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c" Oct 30 00:37:24.416180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.416157 4433 scope.go:122] "RemoveContainer" containerID="23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c" Oct 30 00:37:24.416502 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:24.416481 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c\": container with ID starting with 23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c not found: ID does not exist" containerID="23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c" Oct 30 00:37:24.416548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.416511 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c"} err="failed to get container status \"23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c\": rpc error: code = NotFound desc = could not find container \"23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c\": container with ID starting with 23953da3993145c57ff9495bfebf2f0c8827652632fd5ff3b0d065e6abd62b7c not found: ID does not exist" Oct 30 00:37:24.440473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.440432 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" podStartSLOduration=2.440419525 podStartE2EDuration="2.440419525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:24.439954151 +0000 UTC m=+1953.374022707" watchObservedRunningTime="2026-10-30 00:37:24.440419525 +0000 UTC m=+1953.374488102" Oct 30 00:37:24.484126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.484086 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k"] Oct 30 00:37:24.487390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:24.487362 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c1eeb6/auth-test-model-c1eeb6-kserve-58545675cc-rf49k"] Oct 30 00:37:25.185408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:25.185374 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" path="/var/lib/kubelet/pods/4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77/volumes" Oct 30 00:37:25.248552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:25.248515 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:25.250445 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:25.250415 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:37:25.250625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:25.250473 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 00:37:25.794141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:25.794107 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:26.248834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:26.248798 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:37:26.249300 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:26.248854 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 00:37:27.248510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:27.248478 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:37:27.248717 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:27.248538 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 00:37:27.313902 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:27.313868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:27.474592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:27.471755 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4"] Oct 30 00:37:27.504488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:27.504210 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:27.628597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:27.620204 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" containerID="cri-o://5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0" gracePeriod=45 Oct 30 00:37:28.007274 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.007246 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:37:28.033085 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033056 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93a188e-e62f-4e56-b197-87d5361ab3ea-kube-api-access-mbc5t\" (UniqueName: \"kubernetes.io/projected/b93a188e-e62f-4e56-b197-87d5361ab3ea-kube-api-access-mbc5t\") pod \"b93a188e-e62f-4e56-b197-87d5361ab3ea\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " Oct 30 00:37:28.033275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033182 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-dshm\") pod \"b93a188e-e62f-4e56-b197-87d5361ab3ea\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " Oct 30 00:37:28.033275 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033248 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-model-cache\") pod \"b93a188e-e62f-4e56-b197-87d5361ab3ea\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " Oct 30 00:37:28.033403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033279 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-tmp-dir\") pod \"b93a188e-e62f-4e56-b197-87d5361ab3ea\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " Oct 30 00:37:28.033403 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033386 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-home\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-home\") pod \"b93a188e-e62f-4e56-b197-87d5361ab3ea\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " Oct 30 00:37:28.033519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033505 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93a188e-e62f-4e56-b197-87d5361ab3ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/b93a188e-e62f-4e56-b197-87d5361ab3ea-tls-certs\") pod \"b93a188e-e62f-4e56-b197-87d5361ab3ea\" (UID: \"b93a188e-e62f-4e56-b197-87d5361ab3ea\") " Oct 30 00:37:28.033591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033520 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-model-cache" pod "b93a188e-e62f-4e56-b197-87d5361ab3ea" (UID: "b93a188e-e62f-4e56-b197-87d5361ab3ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.033671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033651 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-tmp-dir" pod "b93a188e-e62f-4e56-b197-87d5361ab3ea" (UID: "b93a188e-e62f-4e56-b197-87d5361ab3ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.033735 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.033672 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-home" pod "b93a188e-e62f-4e56-b197-87d5361ab3ea" (UID: "b93a188e-e62f-4e56-b197-87d5361ab3ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.034122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.034099 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.034210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.034125 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.034210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.034140 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.035635 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.035610 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93a188e-e62f-4e56-b197-87d5361ab3ea-tls-certs" pod "b93a188e-e62f-4e56-b197-87d5361ab3ea" (UID: "b93a188e-e62f-4e56-b197-87d5361ab3ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:28.035760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.035699 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93a188e-e62f-4e56-b197-87d5361ab3ea-kube-api-access-mbc5t" pod "b93a188e-e62f-4e56-b197-87d5361ab3ea" (UID: "b93a188e-e62f-4e56-b197-87d5361ab3ea"). InnerVolumeSpecName "kube-api-access-mbc5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:28.035760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.035718 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-dshm" pod "b93a188e-e62f-4e56-b197-87d5361ab3ea" (UID: "b93a188e-e62f-4e56-b197-87d5361ab3ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.138151 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.138019 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93a188e-e62f-4e56-b197-87d5361ab3ea-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.138151 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.138057 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93a188e-e62f-4e56-b197-87d5361ab3ea-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.138151 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.138074 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbc5t\" (UniqueName: \"kubernetes.io/projected/b93a188e-e62f-4e56-b197-87d5361ab3ea-kube-api-access-mbc5t\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.250228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.248552 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:37:28.250228 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.248656 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 00:37:28.426689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.426596 4433 generic.go:356] "Generic (PLEG): container finished" podID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerID="5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0" exitCode=0 Oct 30 00:37:28.426689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.426651 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" Oct 30 00:37:28.427180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.426684 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" event={"ID":"b93a188e-e62f-4e56-b197-87d5361ab3ea","Type":"ContainerDied","Data":"5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0"} Oct 30 00:37:28.427180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.426729 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr" event={"ID":"b93a188e-e62f-4e56-b197-87d5361ab3ea","Type":"ContainerDied","Data":"8de3c4dd5b350a7efb56e87f4484e5c256c847e6b45c0099c28fa1dd3991fd9a"} Oct 30 00:37:28.427180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.426747 4433 scope.go:122] "RemoveContainer" containerID="5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0" Oct 30 00:37:28.441600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.441560 4433 scope.go:122] "RemoveContainer" containerID="5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0" Oct 30 00:37:28.441967 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:28.441937 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0\": container with ID starting with 5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0 not found: ID does not exist" containerID="5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0" Oct 30 00:37:28.442043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.441981 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0"} err="failed to get container status \"5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0\": rpc error: code = NotFound desc = could not find container \"5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0\": container with ID starting with 5f53e8bcbad19159690a21ed5ad96c53d0cab8cbf100d252b939821812c94ee0 not found: ID does not exist" Oct 30 00:37:28.498360 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.498321 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr"] Oct 30 00:37:28.501941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:28.501910 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/simulator-w1-7a7ac6-kserve-7fdf4c7475-dczqr"] Oct 30 00:37:29.129419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:29.129389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:29.199282 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:29.199248 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" path="/var/lib/kubelet/pods/b93a188e-e62f-4e56-b197-87d5361ab3ea/volumes" Oct 30 00:37:29.248005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:29.247963 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:37:29.248187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:29.248031 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 00:37:30.253381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:30.253346 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:30.253823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:30.253780 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:30.254417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:30.254392 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:37:30.801005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:30.800961 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:32.277036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:32.276989 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:33.776814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:33.776774 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:35.385376 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:35.385319 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:36.976149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:36.976101 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:37.115493 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:37.115451 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:37:38.318756 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:38.318709 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:38.425839 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:38.425791 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:38.843064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:38.843028 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj"] Oct 30 00:37:38.843308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:38.843280 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" containerID="cri-o://d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d" gracePeriod=30 Oct 30 00:37:38.985022 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:38.984983 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:39.434639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:39.434599 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh"] Oct 30 00:37:39.465636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:39.465607 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:39.587434 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:39.587395 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:39.689366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:39.689281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:40.176825 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:40.176793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:40.283616 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:40.283544 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:40.794493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:40.794454 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:42.453223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.453190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:42.528654 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.528609 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" podUID="4a999892-3977-46db-89f3-1cf2f172a2e5" containerName="main" containerID="cri-o://592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade" gracePeriod=45 Oct 30 00:37:42.827849 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.827820 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:37:42.952003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.951974 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-dshm\") pod \"4a999892-3977-46db-89f3-1cf2f172a2e5\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " Oct 30 00:37:42.952201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952092 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-home\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-home\") pod \"4a999892-3977-46db-89f3-1cf2f172a2e5\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " Oct 30 00:37:42.952201 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952154 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-model-cache\") pod \"4a999892-3977-46db-89f3-1cf2f172a2e5\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " Oct 30 00:37:42.952331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952236 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a999892-3977-46db-89f3-1cf2f172a2e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a999892-3977-46db-89f3-1cf2f172a2e5-tls-certs\") pod \"4a999892-3977-46db-89f3-1cf2f172a2e5\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " Oct 30 00:37:42.952331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952279 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a999892-3977-46db-89f3-1cf2f172a2e5-kube-api-access-hd2nd\" (UniqueName: \"kubernetes.io/projected/4a999892-3977-46db-89f3-1cf2f172a2e5-kube-api-access-hd2nd\") pod \"4a999892-3977-46db-89f3-1cf2f172a2e5\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " Oct 30 00:37:42.952446 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952334 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-tmp-dir\") pod \"4a999892-3977-46db-89f3-1cf2f172a2e5\" (UID: \"4a999892-3977-46db-89f3-1cf2f172a2e5\") " Oct 30 00:37:42.952446 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952358 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-home" pod "4a999892-3977-46db-89f3-1cf2f172a2e5" (UID: "4a999892-3977-46db-89f3-1cf2f172a2e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:42.952530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952452 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-model-cache" pod "4a999892-3977-46db-89f3-1cf2f172a2e5" (UID: "4a999892-3977-46db-89f3-1cf2f172a2e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:42.952719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.952692 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-tmp-dir" pod "4a999892-3977-46db-89f3-1cf2f172a2e5" (UID: "4a999892-3977-46db-89f3-1cf2f172a2e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:42.953099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.953076 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:42.953155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.953106 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:42.953155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.953122 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:42.954409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.954388 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-dshm" pod "4a999892-3977-46db-89f3-1cf2f172a2e5" (UID: "4a999892-3977-46db-89f3-1cf2f172a2e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:42.954611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.954533 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a999892-3977-46db-89f3-1cf2f172a2e5-tls-certs" pod "4a999892-3977-46db-89f3-1cf2f172a2e5" (UID: "4a999892-3977-46db-89f3-1cf2f172a2e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:42.954611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:42.954550 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a999892-3977-46db-89f3-1cf2f172a2e5-kube-api-access-hd2nd" pod "4a999892-3977-46db-89f3-1cf2f172a2e5" (UID: "4a999892-3977-46db-89f3-1cf2f172a2e5"). InnerVolumeSpecName "kube-api-access-hd2nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:43.054418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.054336 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a999892-3977-46db-89f3-1cf2f172a2e5-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:43.054418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.054366 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a999892-3977-46db-89f3-1cf2f172a2e5-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:43.054418 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.054377 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd2nd\" (UniqueName: \"kubernetes.io/projected/4a999892-3977-46db-89f3-1cf2f172a2e5-kube-api-access-hd2nd\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:43.442292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.442258 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-521715"] Oct 30 00:37:43.442931 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.442909 4433 cpu_manager.go:414] "containerMap: removing container" podUID="4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" containerName="main" Oct 30 00:37:43.442989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.442932 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" containerName="main" Oct 30 00:37:43.442989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.442949 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" Oct 30 00:37:43.442989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.442954 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" Oct 30 00:37:43.442989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.442965 4433 cpu_manager.go:414] "containerMap: removing container" podUID="4a999892-3977-46db-89f3-1cf2f172a2e5" containerName="main" Oct 30 00:37:43.442989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.442970 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a999892-3977-46db-89f3-1cf2f172a2e5" containerName="main" Oct 30 00:37:43.443140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.443105 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7ba6c4-0c1c-4515-82f1-de9f90cc6a77" containerName="main" Oct 30 00:37:43.443140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.443114 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93a188e-e62f-4e56-b197-87d5361ab3ea" containerName="main" Oct 30 00:37:43.443140 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.443120 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a999892-3977-46db-89f3-1cf2f172a2e5" containerName="main" Oct 30 00:37:43.446989 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.446971 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:43.465617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.465585 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-521715"] Oct 30 00:37:43.474961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.474919 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hr9mw\"" Oct 30 00:37:43.553013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.552982 4433 generic.go:356] "Generic (PLEG): container finished" podID="4a999892-3977-46db-89f3-1cf2f172a2e5" containerID="592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade" exitCode=0 Oct 30 00:37:43.553157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.553036 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" Oct 30 00:37:43.553157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.553055 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" event={"ID":"4a999892-3977-46db-89f3-1cf2f172a2e5","Type":"ContainerDied","Data":"592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade"} Oct 30 00:37:43.553157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.553096 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4" event={"ID":"4a999892-3977-46db-89f3-1cf2f172a2e5","Type":"ContainerDied","Data":"a4083f6346b89e76fd7eec3c622dedcb36df57b9191c4fd876045595b375a106"} Oct 30 00:37:43.553157 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.553113 4433 scope.go:122] "RemoveContainer" containerID="592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade" Oct 30 00:37:43.560779 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.560749 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwrd\" (UniqueName: \"kubernetes.io/projected/99095068-2c79-4c7f-a79b-5c4e8a550872-kube-api-access-nnwrd\") pod \"test-curl-7593-521715\" (UID: \"99095068-2c79-4c7f-a79b-5c4e8a550872\") " pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:43.564555 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.564530 4433 scope.go:122] "RemoveContainer" containerID="592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade" Oct 30 00:37:43.564883 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:43.564861 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade\": container with ID starting with 592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade not found: ID does not exist" containerID="592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade" Oct 30 00:37:43.564953 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.564890 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade"} err="failed to get container status \"592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade\": rpc error: code = NotFound desc = could not find container \"592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade\": container with ID starting with 592516e41c14a69b363709a4962bba2e70b46e9fbb2da80fbf3d9f2f87619ade not found: ID does not exist" Oct 30 00:37:43.607368 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.607334 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4"] Oct 30 00:37:43.610765 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.610743 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ecb34a/auth-test-model-ecb34a-kserve-f548b5745-dfdq4"] Oct 30 00:37:43.662828 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.662799 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwrd\" (UniqueName: \"kubernetes.io/projected/99095068-2c79-4c7f-a79b-5c4e8a550872-kube-api-access-nnwrd\") pod \"test-curl-7593-521715\" (UID: \"99095068-2c79-4c7f-a79b-5c4e8a550872\") " pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:43.687012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.686982 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwrd\" (UniqueName: \"kubernetes.io/projected/99095068-2c79-4c7f-a79b-5c4e8a550872-kube-api-access-nnwrd\") pod \"test-curl-7593-521715\" (UID: \"99095068-2c79-4c7f-a79b-5c4e8a550872\") " pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:43.760028 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.759938 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:43.792410 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:43.791743 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:44.102655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.102118 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-521715"] Oct 30 00:37:44.242712 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.242677 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:37:44.301308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.301267 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:44.400639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.400508 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-data\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.400797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.400685 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-certs\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.400797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.400754 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-kube-api-access-q42dt\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-kube-api-access-q42dt\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.400934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.400834 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d990bb4a-b08e-49de-ba43-2181a0996cea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d990bb4a-b08e-49de-ba43-2181a0996cea-istiod-ca-cert\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.400934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.400877 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-data" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:44.401051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.401011 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-socket\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.401051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.401042 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-credential-socket\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.401168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.401083 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-podinfo\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.401168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.401144 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-certs" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:44.401168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.401166 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-token\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-token\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.401328 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.401185 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-envoy\") pod \"d990bb4a-b08e-49de-ba43-2181a0996cea\" (UID: \"d990bb4a-b08e-49de-ba43-2181a0996cea\") " Oct 30 00:37:44.401677 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.401648 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-credential-socket" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:44.402099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.402075 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.402099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.402101 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.402099 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.402115 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.402370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.402281 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d990bb4a-b08e-49de-ba43-2181a0996cea-istiod-ca-cert" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:44.402621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.402592 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-socket" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:44.403908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.403846 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-token" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:44.404811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.404791 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-envoy" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:44.405057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.405028 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-kube-api-access-q42dt" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "kube-api-access-q42dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:44.406945 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.405952 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-podinfo" pod "d990bb4a-b08e-49de-ba43-2181a0996cea" (UID: "d990bb4a-b08e-49de-ba43-2181a0996cea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:44.422212 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.421013 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:44.503864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.503822 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42dt\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-kube-api-access-q42dt\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.503864 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.503861 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d990bb4a-b08e-49de-ba43-2181a0996cea-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.504390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.503875 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.504390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.503888 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.504390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.503903 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.504390 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.503914 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d990bb4a-b08e-49de-ba43-2181a0996cea-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:44.527682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.527643 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:44.565898 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.565857 4433 generic.go:356] "Generic (PLEG): container finished" podID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerID="d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d" exitCode=0 Oct 30 00:37:44.566189 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.566072 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" event={"ID":"d990bb4a-b08e-49de-ba43-2181a0996cea","Type":"ContainerDied","Data":"d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d"} Oct 30 00:37:44.566189 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.566189 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" event={"ID":"d990bb4a-b08e-49de-ba43-2181a0996cea","Type":"ContainerDied","Data":"4f9bfe064326f06d0fa320bddd22c68922d716f759effd7475eb0a881d6b9fc6"} Oct 30 00:37:44.566338 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.566221 4433 scope.go:122] "RemoveContainer" containerID="d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d" Oct 30 00:37:44.566405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.566351 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj" Oct 30 00:37:44.570628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.570600 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-521715" event={"ID":"99095068-2c79-4c7f-a79b-5c4e8a550872","Type":"ContainerStarted","Data":"4debe7761eafd84c69e93e06244baf2a418d4543a2d74766416637a2e7ce0e63"} Oct 30 00:37:44.587766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.587732 4433 scope.go:122] "RemoveContainer" containerID="d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d" Oct 30 00:37:44.588138 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:44.588118 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d\": container with ID starting with d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d not found: ID does not exist" containerID="d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d" Oct 30 00:37:44.588235 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.588145 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d"} err="failed to get container status \"d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d\": rpc error: code = NotFound desc = could not find container \"d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d\": container with ID starting with d4fac31384504b761643c935477151b14abd584e6f97025313e28c104ed4a93d not found: ID does not exist" Oct 30 00:37:44.624844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.624809 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj"] Oct 30 00:37:44.631625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.631357 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-phppj"] Oct 30 00:37:44.936588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:44.936538 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:45.180967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:45.180924 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a999892-3977-46db-89f3-1cf2f172a2e5" path="/var/lib/kubelet/pods/4a999892-3977-46db-89f3-1cf2f172a2e5/volumes" Oct 30 00:37:45.181634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:45.181611 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" path="/var/lib/kubelet/pods/d990bb4a-b08e-49de-ba43-2181a0996cea/volumes" Oct 30 00:37:46.141119 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:46.141087 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:46.419190 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:46.419111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:46.547597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:46.547281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:46.760387 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:46.760288 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:47.648396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:47.648360 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-521715" event={"ID":"99095068-2c79-4c7f-a79b-5c4e8a550872","Type":"ContainerStarted","Data":"c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44"} Oct 30 00:37:47.690220 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:47.690151 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7593-521715" podStartSLOduration=1.516933195 podStartE2EDuration="4.690134243s" totalImagesPullingTime="3.173201048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:47.687498497 +0000 UTC m=+1976.621567051" watchObservedRunningTime="2026-10-30 00:37:47.690134243 +0000 UTC m=+1976.624202779" Oct 30 00:37:47.873288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:47.873250 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:48.604064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:48.604027 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-521715"] Oct 30 00:37:48.604272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:48.604085 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-521715"] Oct 30 00:37:48.862147 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:48.862060 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:49.327672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:49.327633 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:49.505328 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:49.505294 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:49.673530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:49.669920 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7593-521715" podUID="99095068-2c79-4c7f-a79b-5c4e8a550872" containerName="test-curl-7593-521715" containerID="cri-o://c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44" gracePeriod=2 Oct 30 00:37:50.001686 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:50.001434 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:50.126086 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:50.126054 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:51.430222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:51.430188 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:51.620831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:51.620793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:51.901043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:51.901011 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:51.964177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:51.964085 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99095068-2c79-4c7f-a79b-5c4e8a550872-kube-api-access-nnwrd\" (UniqueName: \"kubernetes.io/projected/99095068-2c79-4c7f-a79b-5c4e8a550872-kube-api-access-nnwrd\") pod \"99095068-2c79-4c7f-a79b-5c4e8a550872\" (UID: \"99095068-2c79-4c7f-a79b-5c4e8a550872\") " Oct 30 00:37:51.966218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:51.966186 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99095068-2c79-4c7f-a79b-5c4e8a550872-kube-api-access-nnwrd" pod "99095068-2c79-4c7f-a79b-5c4e8a550872" (UID: "99095068-2c79-4c7f-a79b-5c4e8a550872"). InnerVolumeSpecName "kube-api-access-nnwrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:52.066101 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.066060 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwrd\" (UniqueName: \"kubernetes.io/projected/99095068-2c79-4c7f-a79b-5c4e8a550872-kube-api-access-nnwrd\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.178530 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.178493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:52.293480 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.293384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:52.498434 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.498395 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx"] Oct 30 00:37:52.499030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.498679 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" containerID="cri-o://25ba1b93d3b26148825a53714bb7e9d0e2ba9938dc9928fc15053df8321d005f" gracePeriod=30 Oct 30 00:37:52.526411 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.526371 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf"] Oct 30 00:37:52.526659 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.526633 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" containerID="cri-o://de1d8d6e7f4b574994a9709ad1ede354d68a6b17d23290fc42860a3bf807daf2" gracePeriod=30 Oct 30 00:37:52.538350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.538268 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:52.539204 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.539183 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z"] Oct 30 00:37:52.540112 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.539361 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" containerID="cri-o://2e451e6cd99d714cd32731fc6e284d9832ba7dffdac0f51f90307ad00ded0fc9" gracePeriod=30 Oct 30 00:37:52.753597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.753372 4433 generic.go:356] "Generic (PLEG): container finished" podID="99095068-2c79-4c7f-a79b-5c4e8a550872" containerID="c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44" exitCode=137 Oct 30 00:37:52.753857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.753634 4433 scope.go:122] "RemoveContainer" containerID="c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44" Oct 30 00:37:52.753857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.753828 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:52.840501 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:52.840466 4433 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-521715\" is forbidden: User \"system:node:ip-10-0-138-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-81.ec2.internal' and this object" podUID="99095068-2c79-4c7f-a79b-5c4e8a550872" pod="opendatahub/test-curl-7593-521715" Oct 30 00:37:52.848606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.848201 4433 scope.go:122] "RemoveContainer" containerID="c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44" Oct 30 00:37:52.848725 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:37:52.848653 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44\": container with ID starting with c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44 not found: ID does not exist" containerID="c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44" Oct 30 00:37:52.848725 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:52.848693 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44"} err="failed to get container status \"c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44\": rpc error: code = NotFound desc = could not find container \"c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44\": container with ID starting with c6204ad720948305476a099e6dad26073dac9595a5aa867e656ff668f7cc9c44 not found: ID does not exist" Oct 30 00:37:53.169874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:53.169838 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:53.180056 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:53.179997 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99095068-2c79-4c7f-a79b-5c4e8a550872" path="/var/lib/kubelet/pods/99095068-2c79-4c7f-a79b-5c4e8a550872/volumes" Oct 30 00:37:54.458382 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.458345 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" podUID="2f92ed7a-b83b-47e0-8778-573e8f1d6efc" containerName="main" containerID="cri-o://32c581951afa1fc5878a3f9b9c74d5a8d3bdc02781ebe7eda15917ed6769a7e5" gracePeriod=45 Oct 30 00:37:54.608912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.608868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:54.769558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.769473 4433 generic.go:356] "Generic (PLEG): container finished" podID="2f92ed7a-b83b-47e0-8778-573e8f1d6efc" containerID="32c581951afa1fc5878a3f9b9c74d5a8d3bdc02781ebe7eda15917ed6769a7e5" exitCode=0 Oct 30 00:37:54.769558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.769539 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" event={"ID":"2f92ed7a-b83b-47e0-8778-573e8f1d6efc","Type":"ContainerDied","Data":"32c581951afa1fc5878a3f9b9c74d5a8d3bdc02781ebe7eda15917ed6769a7e5"} Oct 30 00:37:54.818511 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.818481 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:37:54.827684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.827658 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-dshm\") pod \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " Oct 30 00:37:54.827792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.827707 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-home\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-home\") pod \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " Oct 30 00:37:54.827792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.827736 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-model-cache\") pod \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " Oct 30 00:37:54.827792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.827758 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tmp-dir\") pod \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " Oct 30 00:37:54.827792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.827787 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-kube-api-access-b4xnz\" (UniqueName: \"kubernetes.io/projected/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-kube-api-access-b4xnz\") pod \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " Oct 30 00:37:54.827979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.827855 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tls-certs\") pod \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\" (UID: \"2f92ed7a-b83b-47e0-8778-573e8f1d6efc\") " Oct 30 00:37:54.828039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.827998 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-home" pod "2f92ed7a-b83b-47e0-8778-573e8f1d6efc" (UID: "2f92ed7a-b83b-47e0-8778-573e8f1d6efc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:54.828039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.828012 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-model-cache" pod "2f92ed7a-b83b-47e0-8778-573e8f1d6efc" (UID: "2f92ed7a-b83b-47e0-8778-573e8f1d6efc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:54.828039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.828022 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tmp-dir" pod "2f92ed7a-b83b-47e0-8778-573e8f1d6efc" (UID: "2f92ed7a-b83b-47e0-8778-573e8f1d6efc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:54.828628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.828607 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:54.828628 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.828627 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:54.828802 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.828636 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:54.830058 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.830037 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-dshm" pod "2f92ed7a-b83b-47e0-8778-573e8f1d6efc" (UID: "2f92ed7a-b83b-47e0-8778-573e8f1d6efc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:54.830150 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.830055 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tls-certs" pod "2f92ed7a-b83b-47e0-8778-573e8f1d6efc" (UID: "2f92ed7a-b83b-47e0-8778-573e8f1d6efc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:54.830196 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.830157 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-kube-api-access-b4xnz" pod "2f92ed7a-b83b-47e0-8778-573e8f1d6efc" (UID: "2f92ed7a-b83b-47e0-8778-573e8f1d6efc"). InnerVolumeSpecName "kube-api-access-b4xnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:54.930184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.930148 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:54.930184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.930177 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4xnz\" (UniqueName: \"kubernetes.io/projected/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-kube-api-access-b4xnz\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:54.930184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:54.930187 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f92ed7a-b83b-47e0-8778-573e8f1d6efc-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:55.777652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:55.777616 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" Oct 30 00:37:55.777652 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:55.777643 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh" event={"ID":"2f92ed7a-b83b-47e0-8778-573e8f1d6efc","Type":"ContainerDied","Data":"ca5c6b17664aedc335e0cb372408ef9c3a2165e16c87e987188044010d997322"} Oct 30 00:37:55.778160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:55.777695 4433 scope.go:122] "RemoveContainer" containerID="32c581951afa1fc5878a3f9b9c74d5a8d3bdc02781ebe7eda15917ed6769a7e5" Oct 30 00:37:55.833184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:55.833108 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh"] Oct 30 00:37:55.836755 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:55.836737 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-29f3ca/auth-test-model-29f3ca-kserve-5b8b754fbf-9vnmh"] Oct 30 00:37:56.191492 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:56.191460 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:57.179937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.179903 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f92ed7a-b83b-47e0-8778-573e8f1d6efc" path="/var/lib/kubelet/pods/2f92ed7a-b83b-47e0-8778-573e8f1d6efc/volumes" Oct 30 00:37:57.710219 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.710179 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:57.833592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.831319 4433 generic.go:356] "Generic (PLEG): container finished" podID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerID="2e451e6cd99d714cd32731fc6e284d9832ba7dffdac0f51f90307ad00ded0fc9" exitCode=0 Oct 30 00:37:57.833592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.831500 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" event={"ID":"4c6ca814-0703-421b-afdd-dfe9b05555ea","Type":"ContainerDied","Data":"2e451e6cd99d714cd32731fc6e284d9832ba7dffdac0f51f90307ad00ded0fc9"} Oct 30 00:37:57.833851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.833701 4433 generic.go:356] "Generic (PLEG): container finished" podID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerID="25ba1b93d3b26148825a53714bb7e9d0e2ba9938dc9928fc15053df8321d005f" exitCode=0 Oct 30 00:37:57.840211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.833930 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" event={"ID":"735ced8b-6586-4e93-8d8a-3439fab15f89","Type":"ContainerDied","Data":"25ba1b93d3b26148825a53714bb7e9d0e2ba9938dc9928fc15053df8321d005f"} Oct 30 00:37:57.840211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.837310 4433 generic.go:356] "Generic (PLEG): container finished" podID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerID="de1d8d6e7f4b574994a9709ad1ede354d68a6b17d23290fc42860a3bf807daf2" exitCode=0 Oct 30 00:37:57.840211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.837385 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" event={"ID":"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa","Type":"ContainerDied","Data":"de1d8d6e7f4b574994a9709ad1ede354d68a6b17d23290fc42860a3bf807daf2"} Oct 30 00:37:57.907256 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.907229 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:37:57.920923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.920834 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:57.934147 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:57.934117 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000375 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-credential-socket\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000474 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-envoy\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000501 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-data\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000556 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-socket\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000605 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-certs\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000654 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-kube-api-access-mjb5m\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-kube-api-access-mjb5m\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000726 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-envoy\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000763 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-credential-socket" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000775 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istiod-ca-cert\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.000972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000899 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-certs\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.000988 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-token\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-token\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001075 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-token\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-token\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001122 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-podinfo\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001161 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-data\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001166 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-certs" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001253 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735ced8b-6586-4e93-8d8a-3439fab15f89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735ced8b-6586-4e93-8d8a-3439fab15f89-istiod-ca-cert\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001313 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-socket\") pod \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\" (UID: \"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001361 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-data" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001401 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-credential-socket\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001474 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-kube-api-access-6t7vq\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-kube-api-access-6t7vq\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001536 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-podinfo\") pod \"735ced8b-6586-4e93-8d8a-3439fab15f89\" (UID: \"735ced8b-6586-4e93-8d8a-3439fab15f89\") " Oct 30 00:37:58.001827 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001648 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-socket" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.002420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.001832 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-data" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.002420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.002384 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.002420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.002406 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.002516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.002419 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.002516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.002436 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.002516 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.002449 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.002707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.002683 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-certs" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.002845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.002825 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istiod-ca-cert" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:58.004144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.004103 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-socket" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.004266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.004159 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735ced8b-6586-4e93-8d8a-3439fab15f89-istiod-ca-cert" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:58.004396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.004375 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-credential-socket" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.006241 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.006211 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-token" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.006468 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.006444 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-envoy" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.009680 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.009636 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-kube-api-access-6t7vq" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "kube-api-access-6t7vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.009798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.009777 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-token" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.009904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.009880 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-podinfo" pod "735ced8b-6586-4e93-8d8a-3439fab15f89" (UID: "735ced8b-6586-4e93-8d8a-3439fab15f89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:58.010019 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.009991 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-envoy" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.011489 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.011430 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-kube-api-access-mjb5m" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "kube-api-access-mjb5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.011872 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.011848 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-podinfo" pod "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" (UID: "191e9810-53a6-41a4-b6fd-aa2f8b9a5faa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:58.041171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.040824 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:58.099499 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.099434 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:37:58.104288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.104261 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735ced8b-6586-4e93-8d8a-3439fab15f89-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.104456 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.104441 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.104567 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.104552 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.104759 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.104743 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t7vq\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-kube-api-access-6t7vq\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.104858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.104847 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.104942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.104930 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.105029 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.105018 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjb5m\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-kube-api-access-mjb5m\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.105122 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.105110 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.105211 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.105202 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.105293 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.105284 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.105369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.105360 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.105444 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.105435 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735ced8b-6586-4e93-8d8a-3439fab15f89-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.105519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.105509 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.207000 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.206965 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-certs\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207089 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-podinfo\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207155 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-kube-api-access-gbl82\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-kube-api-access-gbl82\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207185 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-envoy\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207234 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6ca814-0703-421b-afdd-dfe9b05555ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6ca814-0703-421b-afdd-dfe9b05555ea-istiod-ca-cert\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207275 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-certs" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207358 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-token\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-token\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207406 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-data\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.207495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207487 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-credential-socket\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.208077 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.207597 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-socket\") pod \"4c6ca814-0703-421b-afdd-dfe9b05555ea\" (UID: \"4c6ca814-0703-421b-afdd-dfe9b05555ea\") " Oct 30 00:37:58.208138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.208089 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.208435 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.208404 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-data" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.208538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.208508 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-credential-socket" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.208747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.208726 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-socket" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.209155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.209129 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6ca814-0703-421b-afdd-dfe9b05555ea-istiod-ca-cert" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:58.210043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.210014 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-kube-api-access-gbl82" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "kube-api-access-gbl82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.210631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.210605 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-token" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.211062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.211037 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-envoy" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.211689 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.211666 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-podinfo" pod "4c6ca814-0703-421b-afdd-dfe9b05555ea" (UID: "4c6ca814-0703-421b-afdd-dfe9b05555ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:58.310033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.309994 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbl82\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-kube-api-access-gbl82\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.310033 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.310032 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.310272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.310045 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6ca814-0703-421b-afdd-dfe9b05555ea-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.310272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.310057 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.310272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.310068 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.310272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.310079 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.310272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.310091 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6ca814-0703-421b-afdd-dfe9b05555ea-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.310272 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.310103 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6ca814-0703-421b-afdd-dfe9b05555ea-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.846050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.846018 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" Oct 30 00:37:58.846050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.846045 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf" event={"ID":"191e9810-53a6-41a4-b6fd-aa2f8b9a5faa","Type":"ContainerDied","Data":"72ac1f5c238102320a5a9be63d42cff1bbe8fa7d7c9a1c036e64e25d4a2afd1e"} Oct 30 00:37:58.846336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.846113 4433 scope.go:122] "RemoveContainer" containerID="de1d8d6e7f4b574994a9709ad1ede354d68a6b17d23290fc42860a3bf807daf2" Oct 30 00:37:58.847550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.847531 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" Oct 30 00:37:58.847550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.847534 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z" event={"ID":"4c6ca814-0703-421b-afdd-dfe9b05555ea","Type":"ContainerDied","Data":"1c67a7808615bd3c50824394b7bb3f4df0e87f9ade753dd5704b1df4aece5e91"} Oct 30 00:37:58.849222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.849185 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" Oct 30 00:37:58.849326 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.849193 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx" event={"ID":"735ced8b-6586-4e93-8d8a-3439fab15f89","Type":"ContainerDied","Data":"7fa2293c0d4e1ea5d9b9317eba12206bc892e3872dc9476a8daabcfd441d7fc4"} Oct 30 00:37:58.859857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.859833 4433 scope.go:122] "RemoveContainer" containerID="2e451e6cd99d714cd32731fc6e284d9832ba7dffdac0f51f90307ad00ded0fc9" Oct 30 00:37:58.869223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.869196 4433 scope.go:122] "RemoveContainer" containerID="25ba1b93d3b26148825a53714bb7e9d0e2ba9938dc9928fc15053df8321d005f" Oct 30 00:37:58.917533 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.917500 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf"] Oct 30 00:37:58.928646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.928614 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-qljjf"] Oct 30 00:37:58.955544 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.955512 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx"] Oct 30 00:37:58.958747 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.958722 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-gf4bx"] Oct 30 00:37:58.994464 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.994432 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z"] Oct 30 00:37:58.999143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:58.999112 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-lc84z"] Oct 30 00:37:59.181640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:59.181539 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" path="/var/lib/kubelet/pods/191e9810-53a6-41a4-b6fd-aa2f8b9a5faa/volumes" Oct 30 00:37:59.182109 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:59.182091 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" path="/var/lib/kubelet/pods/4c6ca814-0703-421b-afdd-dfe9b05555ea/volumes" Oct 30 00:37:59.182541 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:59.182524 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" path="/var/lib/kubelet/pods/735ced8b-6586-4e93-8d8a-3439fab15f89/volumes" Oct 30 00:37:59.297439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:37:59.297407 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:00.133300 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:00.133264 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:00.275598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:00.271874 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:00.592215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:00.592181 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:01.384332 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:01.384289 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:02.694795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:02.694750 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:03.259599 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:03.259553 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:03.362250 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:03.362214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:03.481655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:03.481560 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:03.908290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:03.908252 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:05.089630 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:05.089568 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:05.844678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:05.844633 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:05.965746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:05.965713 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:06.088976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:06.088931 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:06.816861 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:06.816828 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:06.930215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:06.928271 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:07.100405 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:07.100328 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:38:08.281391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:08.281355 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:09.894161 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:09.894122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:10.020597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:10.018984 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:10.266143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:10.266060 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:10.766144 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:10.766110 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:11.882869 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:11.882827 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:13.621704 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:13.621662 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:15.368719 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:15.368680 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:15.726993 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:15.726880 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb"] Oct 30 00:38:15.740259 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:15.740227 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq"] Oct 30 00:38:15.820180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:15.820145 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:16.922822 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:16.922785 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:17.032660 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:17.032628 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:18.782126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:18.782092 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:18.896003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:18.895959 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:19.409012 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:19.408978 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:20.796794 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:20.796757 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:21.622617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:21.616734 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:21.741174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:21.741127 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:21.902229 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:21.902142 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:22.215215 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:22.215124 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:22.845943 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:22.845907 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:22.954030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:22.953975 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:23.056932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:23.056895 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:23.491401 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:23.491360 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:24.782694 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:24.782655 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:25.919879 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:25.919846 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:26.043874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:26.043843 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:26.316504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:26.316475 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:26.513328 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:26.513293 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:28.197682 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:28.197638 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:30.038289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:30.038250 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:30.749271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:30.749232 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" containerID="cri-o://d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b" gracePeriod=45 Oct 30 00:38:30.761771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:30.761742 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:30.764013 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:30.763982 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" containerID="cri-o://1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba" gracePeriod=45 Oct 30 00:38:30.882224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:30.882194 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:30.967676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:30.967638 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Oct 30 00:38:31.082610 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.079922 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:38:31.088916 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.088891 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:38:31.106312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.106281 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn"] Oct 30 00:38:31.106979 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.106553 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" containerID="cri-o://3987de7ed327b26ec42aebf970bb47a2970b64633a1a0ba2620a16e4069b9ed1" gracePeriod=30 Oct 30 00:38:31.139224 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.137878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:31.178676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.178613 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-home\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-home\") pod \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " Oct 30 00:38:31.178676 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.178673 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-model-cache\") pod \"89f65f05-73d3-4d40-81f7-3300f86869ef\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " Oct 30 00:38:31.178947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.178727 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-model-cache\") pod \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " Oct 30 00:38:31.178947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.178772 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-home\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-home\") pod \"89f65f05-73d3-4d40-81f7-3300f86869ef\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " Oct 30 00:38:31.178947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.178817 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-kube-api-access-2jdmj\" (UniqueName: \"kubernetes.io/projected/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-kube-api-access-2jdmj\") pod \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " Oct 30 00:38:31.179133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.178963 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tls-certs\") pod \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " Oct 30 00:38:31.179133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.179070 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-tmp-dir\") pod \"89f65f05-73d3-4d40-81f7-3300f86869ef\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " Oct 30 00:38:31.179133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.179102 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f65f05-73d3-4d40-81f7-3300f86869ef-kube-api-access-fjbkd\" (UniqueName: \"kubernetes.io/projected/89f65f05-73d3-4d40-81f7-3300f86869ef-kube-api-access-fjbkd\") pod \"89f65f05-73d3-4d40-81f7-3300f86869ef\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " Oct 30 00:38:31.179289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.179143 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-dshm\") pod \"89f65f05-73d3-4d40-81f7-3300f86869ef\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " Oct 30 00:38:31.179289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.179213 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-dshm\") pod \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " Oct 30 00:38:31.179289 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.179242 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tmp-dir\") pod \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\" (UID: \"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d\") " Oct 30 00:38:31.179449 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.179345 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f65f05-73d3-4d40-81f7-3300f86869ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/89f65f05-73d3-4d40-81f7-3300f86869ef-tls-certs\") pod \"89f65f05-73d3-4d40-81f7-3300f86869ef\" (UID: \"89f65f05-73d3-4d40-81f7-3300f86869ef\") " Oct 30 00:38:31.181173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.179795 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-model-cache" pod "89f65f05-73d3-4d40-81f7-3300f86869ef" (UID: "89f65f05-73d3-4d40-81f7-3300f86869ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.181173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.180035 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.181173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.180275 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-home" pod "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" (UID: "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.181173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.180314 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-model-cache" pod "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" (UID: "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.181173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.180323 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-tmp-dir" pod "89f65f05-73d3-4d40-81f7-3300f86869ef" (UID: "89f65f05-73d3-4d40-81f7-3300f86869ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.181173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.180849 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tmp-dir" pod "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" (UID: "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.181173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.181097 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-home" pod "89f65f05-73d3-4d40-81f7-3300f86869ef" (UID: "89f65f05-73d3-4d40-81f7-3300f86869ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.183645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.183314 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f65f05-73d3-4d40-81f7-3300f86869ef-tls-certs" pod "89f65f05-73d3-4d40-81f7-3300f86869ef" (UID: "89f65f05-73d3-4d40-81f7-3300f86869ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:31.185649 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.185564 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tls-certs" pod "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" (UID: "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:31.185783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.185743 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-dshm" pod "89f65f05-73d3-4d40-81f7-3300f86869ef" (UID: "89f65f05-73d3-4d40-81f7-3300f86869ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.185905 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.185883 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-kube-api-access-2jdmj" pod "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" (UID: "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d"). InnerVolumeSpecName "kube-api-access-2jdmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:31.186475 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.186441 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f65f05-73d3-4d40-81f7-3300f86869ef-kube-api-access-fjbkd" pod "89f65f05-73d3-4d40-81f7-3300f86869ef" (UID: "89f65f05-73d3-4d40-81f7-3300f86869ef"). InnerVolumeSpecName "kube-api-access-fjbkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:31.187814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.187788 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-dshm" pod "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" (UID: "dfb6c7a9-f3dd-4f0d-8081-d274df193a9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.238285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.238246 4433 generic.go:356] "Generic (PLEG): container finished" podID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerID="1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba" exitCode=0 Oct 30 00:38:31.238473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.238296 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" Oct 30 00:38:31.238473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.238323 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" event={"ID":"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d","Type":"ContainerDied","Data":"1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba"} Oct 30 00:38:31.238473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.238384 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq" event={"ID":"dfb6c7a9-f3dd-4f0d-8081-d274df193a9d","Type":"ContainerDied","Data":"3e4968eeb638532cc0613fe19793ca4daac82de50973c515642f9845eb730d72"} Oct 30 00:38:31.238473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.238406 4433 scope.go:122] "RemoveContainer" containerID="1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba" Oct 30 00:38:31.251219 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.251187 4433 generic.go:356] "Generic (PLEG): container finished" podID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerID="d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b" exitCode=0 Oct 30 00:38:31.251350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.251274 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" event={"ID":"89f65f05-73d3-4d40-81f7-3300f86869ef","Type":"ContainerDied","Data":"d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b"} Oct 30 00:38:31.251350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.251310 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" event={"ID":"89f65f05-73d3-4d40-81f7-3300f86869ef","Type":"ContainerDied","Data":"7f9a58e8b31dee5cf4826e14750b82b047320163090e88eef5d81e47ea1c7361"} Oct 30 00:38:31.251473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.251373 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" Oct 30 00:38:31.260974 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.260952 4433 scope.go:122] "RemoveContainer" containerID="1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba" Oct 30 00:38:31.261306 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:31.261278 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba\": container with ID starting with 1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba not found: ID does not exist" containerID="1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba" Oct 30 00:38:31.261370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.261320 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba"} err="failed to get container status \"1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba\": rpc error: code = NotFound desc = could not find container \"1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba\": container with ID starting with 1fda18d088930e29c1c2e460fa74ae75b1f889d38d8a5b08d7ae359461c84aba not found: ID does not exist" Oct 30 00:38:31.261370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.261348 4433 scope.go:122] "RemoveContainer" containerID="d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b" Oct 30 00:38:31.272266 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.272242 4433 scope.go:122] "RemoveContainer" containerID="d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b" Oct 30 00:38:31.272618 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:31.272555 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b\": container with ID starting with d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b not found: ID does not exist" containerID="d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b" Oct 30 00:38:31.272734 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.272632 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b"} err="failed to get container status \"d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b\": rpc error: code = NotFound desc = could not find container \"d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b\": container with ID starting with d6db5b8a995b2eb658db1b77767b72c4c8b663292e888033c277593d8b5dbe5b not found: ID does not exist" Oct 30 00:38:31.281588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281514 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281544 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281588 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281558 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281611 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f65f05-73d3-4d40-81f7-3300f86869ef-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281626 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281637 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281649 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281662 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdmj\" (UniqueName: \"kubernetes.io/projected/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-kube-api-access-2jdmj\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281675 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281688 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f65f05-73d3-4d40-81f7-3300f86869ef-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.281813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.281699 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbkd\" (UniqueName: \"kubernetes.io/projected/89f65f05-73d3-4d40-81f7-3300f86869ef-kube-api-access-fjbkd\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.322730 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.322699 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb"] Oct 30 00:38:31.330540 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.330507 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb"] Oct 30 00:38:31.430597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.413492 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq"] Oct 30 00:38:31.430597 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.418024 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/premium-w4-02665b-kserve-6dd9d6d6f-2z2dq"] Oct 30 00:38:31.795164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.795122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:31.971325 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:31.971283 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-2-w4-02665b-kserve-5855f5dcff-mpzsb" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 00:38:33.181396 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:33.181364 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" path="/var/lib/kubelet/pods/89f65f05-73d3-4d40-81f7-3300f86869ef/volumes" Oct 30 00:38:33.181843 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:33.181828 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" path="/var/lib/kubelet/pods/dfb6c7a9-f3dd-4f0d-8081-d274df193a9d/volumes" Oct 30 00:38:33.402141 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:33.402104 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:34.305606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:34.305540 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:34.946317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:34.946232 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:35.058766 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:35.057890 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:35.548764 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:35.548722 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:35.859296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:35.859256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:36.292352 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.292311 4433 generic.go:356] "Generic (PLEG): container finished" podID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerID="3987de7ed327b26ec42aebf970bb47a2970b64633a1a0ba2620a16e4069b9ed1" exitCode=0 Oct 30 00:38:36.292509 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.292406 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" event={"ID":"643500a6-e1c6-4483-a4f6-ae816b5b8987","Type":"ContainerDied","Data":"3987de7ed327b26ec42aebf970bb47a2970b64633a1a0ba2620a16e4069b9ed1"} Oct 30 00:38:36.403331 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.403303 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:38:36.502855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.502819 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-kube-api-access-6z8th\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-kube-api-access-6z8th\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.502932 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-token\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-token\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.502968 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-socket\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.503038 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-data\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503192 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.503180 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-podinfo\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.503212 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643500a6-e1c6-4483-a4f6-ae816b5b8987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643500a6-e1c6-4483-a4f6-ae816b5b8987-istiod-ca-cert\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503313 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.503278 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-credential-socket\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.503314 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-certs\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.503432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.503351 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-envoy\") pod \"643500a6-e1c6-4483-a4f6-ae816b5b8987\" (UID: \"643500a6-e1c6-4483-a4f6-ae816b5b8987\") " Oct 30 00:38:36.504837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.504803 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-socket" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.504971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.504851 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-data" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.506359 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.506332 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-certs" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.506473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.506451 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-credential-socket" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.506889 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.506869 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643500a6-e1c6-4483-a4f6-ae816b5b8987-istiod-ca-cert" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:36.507156 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.507125 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-kube-api-access-6z8th" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "kube-api-access-6z8th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:36.507265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.507189 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-podinfo" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:36.507265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.507232 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-token" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:36.508461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.508434 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-envoy" pod "643500a6-e1c6-4483-a4f6-ae816b5b8987" (UID: "643500a6-e1c6-4483-a4f6-ae816b5b8987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628290 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628332 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628346 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628359 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8th\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-kube-api-access-6z8th\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628372 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628387 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628400 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628412 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643500a6-e1c6-4483-a4f6-ae816b5b8987-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.634650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.628424 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643500a6-e1c6-4483-a4f6-ae816b5b8987-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.689596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:36.681186 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:37.159136 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:37.158917 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:38:37.300814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:37.300782 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" Oct 30 00:38:37.300814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:37.300811 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn" event={"ID":"643500a6-e1c6-4483-a4f6-ae816b5b8987","Type":"ContainerDied","Data":"e07fe0c3ec8e3d3063a7cbab2c7a18dd8a30cb70b09da9b88beeb37854179aba"} Oct 30 00:38:37.301040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:37.300878 4433 scope.go:122] "RemoveContainer" containerID="3987de7ed327b26ec42aebf970bb47a2970b64633a1a0ba2620a16e4069b9ed1" Oct 30 00:38:37.361175 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:37.361142 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn"] Oct 30 00:38:37.372623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:37.372599 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-s4qbn"] Oct 30 00:38:37.491537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:37.491416 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:37.632967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:37.632914 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:38.152543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:38.152512 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:38.748506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:38.748469 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:38.860155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:38.860119 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:39.063833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:39.063801 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:39.179726 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:39.179692 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" path="/var/lib/kubelet/pods/643500a6-e1c6-4483-a4f6-ae816b5b8987/volumes" Oct 30 00:38:39.256971 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:39.256935 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:39.384671 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:39.384588 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:40.332495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:40.332458 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:40.576037 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:40.575999 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:40.985641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:40.985600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:41.096026 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:41.095994 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:42.291970 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:42.291929 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:42.403209 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:42.403175 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:43.983604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:43.983557 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:45.438051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:45.438017 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:45.593397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:45.593357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:45.708594 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:45.708462 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:45.963731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:45.963644 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p"] Oct 30 00:38:45.964255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:45.964194 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" containerID="cri-o://1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd" gracePeriod=30 Oct 30 00:38:46.006848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:46.006805 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:47.169351 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:47.169310 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:47.444105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:47.444016 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v"] Oct 30 00:38:47.444317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:47.444266 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" containerID="cri-o://7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec" gracePeriod=30 Oct 30 00:38:47.459736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:47.459706 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l"] Oct 30 00:38:47.460166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:47.460134 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" containerID="cri-o://966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7" gracePeriod=30 Oct 30 00:38:47.464810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:47.464692 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4"] Oct 30 00:38:47.465208 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:47.465172 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" containerID="cri-o://6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7" gracePeriod=30 Oct 30 00:38:48.445814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.445773 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw"] Oct 30 00:38:48.446383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.446079 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" podUID="3b5cc49e-a720-4888-ab38-37f7380c3d5f" containerName="maas-api" containerID="cri-o://f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80" gracePeriod=30 Oct 30 00:38:48.509758 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.509716 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:48.738723 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.738696 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:38:48.901288 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.901241 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-metrics-certs\") pod \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " Oct 30 00:38:48.901506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.901409 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-maas-api-tls\") pod \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " Oct 30 00:38:48.911561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.910717 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5cc49e-a720-4888-ab38-37f7380c3d5f-kube-api-access-jkfvs\" (UniqueName: \"kubernetes.io/projected/3b5cc49e-a720-4888-ab38-37f7380c3d5f-kube-api-access-jkfvs\") pod \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " Oct 30 00:38:48.911561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.910842 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5cc49e-a720-4888-ab38-37f7380c3d5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5cc49e-a720-4888-ab38-37f7380c3d5f-openshift-service-ca\") pod \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\" (UID: \"3b5cc49e-a720-4888-ab38-37f7380c3d5f\") " Oct 30 00:38:48.915908 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.915871 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5cc49e-a720-4888-ab38-37f7380c3d5f-openshift-service-ca" pod "3b5cc49e-a720-4888-ab38-37f7380c3d5f" (UID: "3b5cc49e-a720-4888-ab38-37f7380c3d5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:48.941123 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.941090 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-metrics-certs" pod "3b5cc49e-a720-4888-ab38-37f7380c3d5f" (UID: "3b5cc49e-a720-4888-ab38-37f7380c3d5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:48.941342 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.941297 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5cc49e-a720-4888-ab38-37f7380c3d5f-kube-api-access-jkfvs" pod "3b5cc49e-a720-4888-ab38-37f7380c3d5f" (UID: "3b5cc49e-a720-4888-ab38-37f7380c3d5f"). InnerVolumeSpecName "kube-api-access-jkfvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:48.950820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:48.949088 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-maas-api-tls" pod "3b5cc49e-a720-4888-ab38-37f7380c3d5f" (UID: "3b5cc49e-a720-4888-ab38-37f7380c3d5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:49.021397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.021320 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:49.021397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.021351 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkfvs\" (UniqueName: \"kubernetes.io/projected/3b5cc49e-a720-4888-ab38-37f7380c3d5f-kube-api-access-jkfvs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:49.021397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.021361 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5cc49e-a720-4888-ab38-37f7380c3d5f-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:49.021397 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.021370 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5cc49e-a720-4888-ab38-37f7380c3d5f-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:49.089034 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.088993 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:49.196697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.196656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:49.443366 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.443333 4433 generic.go:356] "Generic (PLEG): container finished" podID="3b5cc49e-a720-4888-ab38-37f7380c3d5f" containerID="f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80" exitCode=0 Oct 30 00:38:49.443554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.443381 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" Oct 30 00:38:49.443554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.443415 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" event={"ID":"3b5cc49e-a720-4888-ab38-37f7380c3d5f","Type":"ContainerDied","Data":"f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80"} Oct 30 00:38:49.443554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.443469 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw" event={"ID":"3b5cc49e-a720-4888-ab38-37f7380c3d5f","Type":"ContainerDied","Data":"aaffc00d5e11021702aea58718acf689e22c9e6f2d59c0d1eb1db4e08fe8b2c5"} Oct 30 00:38:49.443554 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.443503 4433 scope.go:122] "RemoveContainer" containerID="f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80" Oct 30 00:38:49.453913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.453695 4433 scope.go:122] "RemoveContainer" containerID="f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80" Oct 30 00:38:49.454163 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:49.453980 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80\": container with ID starting with f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80 not found: ID does not exist" containerID="f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80" Oct 30 00:38:49.454163 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.454007 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80"} err="failed to get container status \"f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80\": rpc error: code = NotFound desc = could not find container \"f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80\": container with ID starting with f1220e13a4576f370a65c8d4f8b4f3fa7b12b83d5ea24afd901cb8c278ff3a80 not found: ID does not exist" Oct 30 00:38:49.500002 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.499970 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw"] Oct 30 00:38:49.511036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.511010 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-415d24-56bf4cd45d-gvggw"] Oct 30 00:38:49.636923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:49.636884 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:50.844961 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:50.844921 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:50.955913 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:50.955879 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:51.075991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.075828 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:51.182158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.182119 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5cc49e-a720-4888-ab38-37f7380c3d5f" path="/var/lib/kubelet/pods/3b5cc49e-a720-4888-ab38-37f7380c3d5f/volumes" Oct 30 00:38:51.252639 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.252611 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:38:51.253655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.253630 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-certs\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.253797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.253740 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f1b969-18ae-48bf-bc75-b859ab445850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f1b969-18ae-48bf-bc75-b859ab445850-istiod-ca-cert\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.253868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.253819 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-socket\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.253926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.253864 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-data\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.253926 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.253907 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-kube-api-access-pwzkn\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-kube-api-access-pwzkn\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.254035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.253862 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-certs" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.254035 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.253946 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-credential-socket\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.254154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254073 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-podinfo\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.254154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254074 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-socket" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.254154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254091 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-data" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.254154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254063 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f1b969-18ae-48bf-bc75-b859ab445850-istiod-ca-cert" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:51.254339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254135 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-token\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-token\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.254339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254181 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-credential-socket" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.254339 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254207 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-envoy\") pod \"d5f1b969-18ae-48bf-bc75-b859ab445850\" (UID: \"d5f1b969-18ae-48bf-bc75-b859ab445850\") " Oct 30 00:38:51.254959 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254935 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.255063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254961 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.255063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254978 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.255063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.254991 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.255063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.255006 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f1b969-18ae-48bf-bc75-b859ab445850-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.256210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.256183 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-kube-api-access-pwzkn" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "kube-api-access-pwzkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:51.256324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.256306 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-envoy" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.256647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.256627 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-token" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:51.256722 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.256647 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-podinfo" pod "d5f1b969-18ae-48bf-bc75-b859ab445850" (UID: "d5f1b969-18ae-48bf-bc75-b859ab445850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:51.356538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.356501 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.356538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.356532 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.356538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.356541 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f1b969-18ae-48bf-bc75-b859ab445850-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.356904 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.356549 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzkn\" (UniqueName: \"kubernetes.io/projected/d5f1b969-18ae-48bf-bc75-b859ab445850-kube-api-access-pwzkn\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.460433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.460347 4433 generic.go:356] "Generic (PLEG): container finished" podID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerID="1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd" exitCode=0 Oct 30 00:38:51.460433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.460395 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" Oct 30 00:38:51.460658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.460427 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" event={"ID":"d5f1b969-18ae-48bf-bc75-b859ab445850","Type":"ContainerDied","Data":"1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd"} Oct 30 00:38:51.460658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.460479 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p" event={"ID":"d5f1b969-18ae-48bf-bc75-b859ab445850","Type":"ContainerDied","Data":"62f6f31c8ab10fa73d4d53f262381ee003ca07cdebe80cbae4dd3ce60d5869cb"} Oct 30 00:38:51.460658 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.460502 4433 scope.go:122] "RemoveContainer" containerID="1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd" Oct 30 00:38:51.471361 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.471344 4433 scope.go:122] "RemoveContainer" containerID="1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd" Oct 30 00:38:51.471661 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:51.471634 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd\": container with ID starting with 1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd not found: ID does not exist" containerID="1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd" Oct 30 00:38:51.471798 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.471666 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd"} err="failed to get container status \"1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd\": rpc error: code = NotFound desc = could not find container \"1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd\": container with ID starting with 1844a154e33752d6c49b0b3776160fb2ae0b8d9745828c01e4ce3980aa9094cd not found: ID does not exist" Oct 30 00:38:51.524700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.524658 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p"] Oct 30 00:38:51.532562 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:51.530558 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-kcf6p"] Oct 30 00:38:52.133463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.133425 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm"] Oct 30 00:38:52.134027 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.133742 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" podUID="58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" containerName="maas-api" containerID="cri-o://3bdf7da0a3b08fbf26fd00c4404e495d78bd3eb4b279ed52e22d1deebef6aca6" gracePeriod=30 Oct 30 00:38:52.192772 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.192739 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:52.474714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.474680 4433 generic.go:356] "Generic (PLEG): container finished" podID="58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" containerID="3bdf7da0a3b08fbf26fd00c4404e495d78bd3eb4b279ed52e22d1deebef6aca6" exitCode=0 Oct 30 00:38:52.474866 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.474846 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" event={"ID":"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad","Type":"ContainerDied","Data":"3bdf7da0a3b08fbf26fd00c4404e495d78bd3eb4b279ed52e22d1deebef6aca6"} Oct 30 00:38:52.632640 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.632246 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:38:52.655700 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.655598 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:52.760451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.759519 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-metrics-certs\") pod \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " Oct 30 00:38:52.760451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.759603 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-openshift-service-ca\") pod \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " Oct 30 00:38:52.760451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.759672 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-kube-api-access-xfhmt\" (UniqueName: \"kubernetes.io/projected/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-kube-api-access-xfhmt\") pod \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " Oct 30 00:38:52.760451 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.759704 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-maas-api-tls\") pod \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\" (UID: \"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad\") " Oct 30 00:38:52.774748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.761419 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-openshift-service-ca" pod "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" (UID: "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:52.782389 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.781192 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-maas-api-tls" pod "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" (UID: "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:52.792168 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.791843 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-kube-api-access-xfhmt" pod "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" (UID: "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad"). InnerVolumeSpecName "kube-api-access-xfhmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:52.801362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.797971 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-metrics-certs" pod "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" (UID: "58bbbf0e-a33c-40b1-82bd-7a73bb9699ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:52.812985 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.812946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:52.847832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.847809 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.860977 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-kube-api-access-f5nd7\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-kube-api-access-f5nd7\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861084 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-envoy\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861136 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-credential-socket\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861176 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-socket\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861282 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istiod-ca-cert\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861405 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-token\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-token\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861448 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-certs\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861485 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-data\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.861551 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-podinfo\") pod \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\" (UID: \"7d12bf49-df66-40a5-b4a6-74163b6a88ed\") " Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.862159 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.862182 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.862199 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhmt\" (UniqueName: \"kubernetes.io/projected/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-kube-api-access-xfhmt\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.862213 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.862524 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-socket" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.862779 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istiod-ca-cert" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.862845 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-certs" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.863075 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-data" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.864618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.863235 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-credential-socket" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.869048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.868820 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-kube-api-access-f5nd7" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "kube-api-access-f5nd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:52.869048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.868839 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-envoy" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.869048 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.868884 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-token" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:52.870067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.870043 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-podinfo" pod "7d12bf49-df66-40a5-b4a6-74163b6a88ed" (UID: "7d12bf49-df66-40a5-b4a6-74163b6a88ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:52.930858 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.930763 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:52.935437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.935416 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:38:52.944258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.944228 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:38:52.962812 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.962777 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-envoy\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.962965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.962849 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-certs\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.962965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.962880 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-kube-api-access-t5cmw\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-kube-api-access-t5cmw\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.962965 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.962934 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-kube-api-access-57chk\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-kube-api-access-57chk\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.962981 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-envoy\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.963139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963011 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-token\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-token\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.963139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963044 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-credential-socket\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.963139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963072 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-token\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-token\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963139 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963108 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-certs" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.963417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963204 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istiod-ca-cert\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.963417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963241 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-podinfo\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.963417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963300 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istiod-ca-cert\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963335 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-socket\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963369 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-podinfo\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963419 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-certs\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963483 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-data\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963533 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-data\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.963701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963626 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-credential-socket\") pod \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\" (UID: \"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d\") " Oct 30 00:38:52.963701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.963668 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-socket\") pod \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\" (UID: \"a3f69cc2-29c2-47f4-8b74-34ace130d3d8\") " Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964310 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964333 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964346 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964359 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964372 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nd7\" (UniqueName: \"kubernetes.io/projected/7d12bf49-df66-40a5-b4a6-74163b6a88ed-kube-api-access-f5nd7\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964385 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964399 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964413 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d12bf49-df66-40a5-b4a6-74163b6a88ed-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964427 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d12bf49-df66-40a5-b4a6-74163b6a88ed-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964439 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964651 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-socket" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.964659 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-socket" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.965045 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istiod-ca-cert" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:52.965655 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.965367 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-credential-socket" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.965755 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-envoy" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.965872 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-kube-api-access-57chk" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "kube-api-access-57chk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.965926 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-kube-api-access-t5cmw" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "kube-api-access-t5cmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.965969 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-data" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.966106 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-certs" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.966162 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-data" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.966195 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istiod-ca-cert" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.966296 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-credential-socket" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.966510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.966338 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-token" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:52.967063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.967004 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-podinfo" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:52.967885 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.967859 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-token" pod "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" (UID: "75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:52.968072 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.968051 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-envoy" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:52.968488 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:52.968466 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-podinfo" pod "a3f69cc2-29c2-47f4-8b74-34ace130d3d8" (UID: "a3f69cc2-29c2-47f4-8b74-34ace130d3d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067174 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067206 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067223 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067237 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067251 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067264 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5cmw\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-kube-api-access-t5cmw\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067275 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57chk\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-kube-api-access-57chk\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067287 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067301 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067314 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067325 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067339 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067353 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f69cc2-29c2-47f4-8b74-34ace130d3d8-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.067504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067365 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.068538 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.067378 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.068744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.068514 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.068744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.068717 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:38:53.189731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.189691 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" path="/var/lib/kubelet/pods/d5f1b969-18ae-48bf-bc75-b859ab445850/volumes" Oct 30 00:38:53.380158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.380075 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:53.484903 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.484868 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" Oct 30 00:38:53.485080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.484896 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm" event={"ID":"58bbbf0e-a33c-40b1-82bd-7a73bb9699ad","Type":"ContainerDied","Data":"1d54a015cd17cb8aa2757ce627ed6d048df12e281aa10ce3eb1315fd8f59d0d7"} Oct 30 00:38:53.485080 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.484973 4433 scope.go:122] "RemoveContainer" containerID="3bdf7da0a3b08fbf26fd00c4404e495d78bd3eb4b279ed52e22d1deebef6aca6" Oct 30 00:38:53.486463 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.486443 4433 generic.go:356] "Generic (PLEG): container finished" podID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerID="6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7" exitCode=0 Oct 30 00:38:53.486537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.486515 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" Oct 30 00:38:53.486604 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.486531 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" event={"ID":"a3f69cc2-29c2-47f4-8b74-34ace130d3d8","Type":"ContainerDied","Data":"6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7"} Oct 30 00:38:53.487284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.486682 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4" event={"ID":"a3f69cc2-29c2-47f4-8b74-34ace130d3d8","Type":"ContainerDied","Data":"0f20eb5620b07378ac4c0ad59bce3f3f4f9bc74368a769bf5b18525b5b6d81fd"} Oct 30 00:38:53.489244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.489220 4433 generic.go:356] "Generic (PLEG): container finished" podID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerID="7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec" exitCode=0 Oct 30 00:38:53.489633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.489278 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" Oct 30 00:38:53.489633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.489318 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" event={"ID":"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d","Type":"ContainerDied","Data":"7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec"} Oct 30 00:38:53.489633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.489384 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v" event={"ID":"75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d","Type":"ContainerDied","Data":"7162bbc38810bfbd0e2a2ac5d5d6f8e31c74b6efebcd94350b8bd6ffe528128f"} Oct 30 00:38:53.491169 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.491144 4433 generic.go:356] "Generic (PLEG): container finished" podID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerID="966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7" exitCode=0 Oct 30 00:38:53.491329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.491251 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" Oct 30 00:38:53.491329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.491287 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" event={"ID":"7d12bf49-df66-40a5-b4a6-74163b6a88ed","Type":"ContainerDied","Data":"966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7"} Oct 30 00:38:53.491329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.491326 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l" event={"ID":"7d12bf49-df66-40a5-b4a6-74163b6a88ed","Type":"ContainerDied","Data":"9a8a8e27bb971da433d55365d7c52f726b9279de8de7c35dc9ccd1f9d20486cb"} Oct 30 00:38:53.524940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.508208 4433 scope.go:122] "RemoveContainer" containerID="6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7" Oct 30 00:38:53.524940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.522707 4433 scope.go:122] "RemoveContainer" containerID="6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7" Oct 30 00:38:53.524940 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:53.523785 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7\": container with ID starting with 6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7 not found: ID does not exist" containerID="6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7" Oct 30 00:38:53.524940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.523832 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7"} err="failed to get container status \"6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7\": rpc error: code = NotFound desc = could not find container \"6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7\": container with ID starting with 6ca312dc30ab572aee18bfc4751fd835df7617a5540a1456e066b72b93e05de7 not found: ID does not exist" Oct 30 00:38:53.524940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.523874 4433 scope.go:122] "RemoveContainer" containerID="7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec" Oct 30 00:38:53.536381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.536351 4433 scope.go:122] "RemoveContainer" containerID="7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec" Oct 30 00:38:53.536790 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:53.536762 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec\": container with ID starting with 7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec not found: ID does not exist" containerID="7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec" Oct 30 00:38:53.536870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.536803 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec"} err="failed to get container status \"7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec\": rpc error: code = NotFound desc = could not find container \"7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec\": container with ID starting with 7ea04ce85896589f14a9a18ab1723bb45c02a6073953c1156623a4911c6650ec not found: ID does not exist" Oct 30 00:38:53.536870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.536828 4433 scope.go:122] "RemoveContainer" containerID="966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7" Oct 30 00:38:53.549083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.549056 4433 scope.go:122] "RemoveContainer" containerID="966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7" Oct 30 00:38:53.549422 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:38:53.549400 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7\": container with ID starting with 966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7 not found: ID does not exist" containerID="966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7" Oct 30 00:38:53.549489 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.549428 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7"} err="failed to get container status \"966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7\": rpc error: code = NotFound desc = could not find container \"966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7\": container with ID starting with 966e56a5da09e8c3fea271de32524b8f5b3601f7c0d617c70df6e48a0e83caf7 not found: ID does not exist" Oct 30 00:38:53.551047 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.551024 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm"] Oct 30 00:38:53.554026 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.554004 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-05c6b2-57cb6fc998-29zwm"] Oct 30 00:38:53.563162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.563120 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:53.589416 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.589386 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v"] Oct 30 00:38:53.595780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.595747 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-txg4v"] Oct 30 00:38:53.632185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.632086 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4"] Oct 30 00:38:53.652607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.636804 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7w8q4"] Oct 30 00:38:53.699617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.697666 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l"] Oct 30 00:38:53.704433 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.704395 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-clq6l"] Oct 30 00:38:53.848014 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:53.847971 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:54.065167 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:54.065123 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:54.271264 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:54.271227 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:54.384017 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:54.383930 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:54.690728 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:54.690701 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:55.179238 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:55.179210 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" path="/var/lib/kubelet/pods/58bbbf0e-a33c-40b1-82bd-7a73bb9699ad/volumes" Oct 30 00:38:55.179697 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:55.179682 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" path="/var/lib/kubelet/pods/75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d/volumes" Oct 30 00:38:55.180096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:55.180084 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" path="/var/lib/kubelet/pods/7d12bf49-df66-40a5-b4a6-74163b6a88ed/volumes" Oct 30 00:38:55.180814 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:55.180798 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" path="/var/lib/kubelet/pods/a3f69cc2-29c2-47f4-8b74-34ace130d3d8/volumes" Oct 30 00:38:55.872706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:55.872672 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:55.980348 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:55.980303 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:56.261785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:56.261694 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:56.431152 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:56.431110 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:56.773298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:56.773263 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:58.057243 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:58.057206 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:59.496459 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:59.496419 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:59.617062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:59.617008 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:59.733568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:59.733525 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:59.835346 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:59.835309 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d"] Oct 30 00:38:59.835647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:59.835611 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" containerID="cri-o://6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f" gracePeriod=30 Oct 30 00:38:59.865822 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:38:59.865793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:00.376409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:00.376375 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:00.389509 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:00.389466 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:39:00.389725 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:00.389540 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:39:00.918499 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:00.918465 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:01.572296 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:01.572256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:01.727172 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:01.727139 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8"] Oct 30 00:39:01.849160 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:01.849065 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:03.206642 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:03.206610 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:04.864523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:04.864481 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:05.125937 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.125874 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:39:05.233317 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233278 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-envoy\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233337 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-kube-api-access-jlnpk\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-kube-api-access-jlnpk\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233393 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-podinfo\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233525 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-certs\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233567 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-token\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233651 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-credential-socket\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233849 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233719 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istiod-ca-cert\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233847 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-data\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.233915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.233885 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-socket\") pod \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\" (UID: \"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499\") " Oct 30 00:39:05.234336 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.234303 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istiod-ca-cert" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:05.234472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.234450 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.234783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.234599 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-credential-socket" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:05.234783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.234688 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-certs" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:05.234783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.234705 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-socket" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:05.234783 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.234767 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-data" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:05.235997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.235972 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-podinfo" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:05.236107 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.236008 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-kube-api-access-jlnpk" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "kube-api-access-jlnpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:05.236182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.236097 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-envoy" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:05.236973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.236955 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-token" pod "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" (UID: "5e5c6a4d-05ab-4ba0-aaa9-c6969a303499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:05.336063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336024 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.336063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336057 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.336063 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336066 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlnpk\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-kube-api-access-jlnpk\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.336310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336075 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.336310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336084 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.336310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336092 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.336310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336100 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.336310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.336107 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:05.338771 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.338741 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw"] Oct 30 00:39:05.358823 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.358790 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx"] Oct 30 00:39:05.452845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.452746 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:05.590813 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.590777 4433 generic.go:356] "Generic (PLEG): container finished" podID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerID="6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f" exitCode=0 Oct 30 00:39:05.591006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.590831 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" Oct 30 00:39:05.591006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.590862 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" event={"ID":"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499","Type":"ContainerDied","Data":"6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f"} Oct 30 00:39:05.591006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.590928 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d" event={"ID":"5e5c6a4d-05ab-4ba0-aaa9-c6969a303499","Type":"ContainerDied","Data":"ed2c49768b4e14a8a055512da4c517580483316833281bff934168f22db6f61b"} Oct 30 00:39:05.591006 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.590960 4433 scope.go:122] "RemoveContainer" containerID="6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f" Oct 30 00:39:05.602938 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.602920 4433 scope.go:122] "RemoveContainer" containerID="7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51" Oct 30 00:39:05.612555 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.612537 4433 scope.go:122] "RemoveContainer" containerID="6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f" Oct 30 00:39:05.612815 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:05.612796 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f\": container with ID starting with 6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f not found: ID does not exist" containerID="6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f" Oct 30 00:39:05.612857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.612823 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f"} err="failed to get container status \"6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f\": rpc error: code = NotFound desc = could not find container \"6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f\": container with ID starting with 6b2dad3eaf9b5aa5f8432be6f10c6d58c8d9f346acf167bec3849f73696bb07f not found: ID does not exist" Oct 30 00:39:05.612857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.612841 4433 scope.go:122] "RemoveContainer" containerID="7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51" Oct 30 00:39:05.613047 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:05.613028 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51\": container with ID starting with 7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51 not found: ID does not exist" containerID="7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51" Oct 30 00:39:05.613088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.613053 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51"} err="failed to get container status \"7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51\": rpc error: code = NotFound desc = could not find container \"7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51\": container with ID starting with 7d471f628759aa6a346ea9a0be9310efb38416ce5f487f3f81bec77b6a3d3d51 not found: ID does not exist" Oct 30 00:39:05.649441 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.649401 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d"] Oct 30 00:39:05.652448 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:05.652419 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-05c6b2-openshift-default-77b4c7ff65-kkp8d"] Oct 30 00:39:06.595623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:06.595566 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:07.100457 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:07.100422 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:39:07.180472 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:07.180437 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" path="/var/lib/kubelet/pods/5e5c6a4d-05ab-4ba0-aaa9-c6969a303499/volumes" Oct 30 00:39:07.588032 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:07.587990 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:07.705596 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:07.704464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:08.077558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:08.077521 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:09.619305 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:09.619268 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:09.772624 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:09.772588 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:11.188791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:11.188756 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:11.299641 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:11.298562 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:11.407271 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:11.407233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:12.563500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:12.563464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:12.909521 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:12.909418 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:14.488459 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:14.488424 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:16.158478 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:16.158445 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:16.748646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:16.748563 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" containerID="cri-o://af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651" gracePeriod=45 Oct 30 00:39:17.016499 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.016440 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:39:17.021400 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021377 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e586cc2-705e-44fb-abaa-dd8993da8faf-kube-api-access-5rv4r\" (UniqueName: \"kubernetes.io/projected/2e586cc2-705e-44fb-abaa-dd8993da8faf-kube-api-access-5rv4r\") pod \"2e586cc2-705e-44fb-abaa-dd8993da8faf\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " Oct 30 00:39:17.021510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021423 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-dshm\") pod \"2e586cc2-705e-44fb-abaa-dd8993da8faf\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " Oct 30 00:39:17.021510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021462 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-model-cache\") pod \"2e586cc2-705e-44fb-abaa-dd8993da8faf\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " Oct 30 00:39:17.021510 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021494 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-home\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-home\") pod \"2e586cc2-705e-44fb-abaa-dd8993da8faf\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " Oct 30 00:39:17.021633 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021618 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-tmp-dir\") pod \"2e586cc2-705e-44fb-abaa-dd8993da8faf\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " Oct 30 00:39:17.021736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021720 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e586cc2-705e-44fb-abaa-dd8993da8faf-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e586cc2-705e-44fb-abaa-dd8993da8faf-tls-certs\") pod \"2e586cc2-705e-44fb-abaa-dd8993da8faf\" (UID: \"2e586cc2-705e-44fb-abaa-dd8993da8faf\") " Oct 30 00:39:17.021857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021835 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-model-cache" pod "2e586cc2-705e-44fb-abaa-dd8993da8faf" (UID: "2e586cc2-705e-44fb-abaa-dd8993da8faf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.021923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021856 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-home" pod "2e586cc2-705e-44fb-abaa-dd8993da8faf" (UID: "2e586cc2-705e-44fb-abaa-dd8993da8faf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.021923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.021875 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-tmp-dir" pod "2e586cc2-705e-44fb-abaa-dd8993da8faf" (UID: "2e586cc2-705e-44fb-abaa-dd8993da8faf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.022090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.022075 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.022090 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.022091 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.022181 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.022099 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.023600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.023567 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-dshm" pod "2e586cc2-705e-44fb-abaa-dd8993da8faf" (UID: "2e586cc2-705e-44fb-abaa-dd8993da8faf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.023725 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.023702 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e586cc2-705e-44fb-abaa-dd8993da8faf-tls-certs" pod "2e586cc2-705e-44fb-abaa-dd8993da8faf" (UID: "2e586cc2-705e-44fb-abaa-dd8993da8faf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:17.023835 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.023816 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e586cc2-705e-44fb-abaa-dd8993da8faf-kube-api-access-5rv4r" pod "2e586cc2-705e-44fb-abaa-dd8993da8faf" (UID: "2e586cc2-705e-44fb-abaa-dd8993da8faf"). InnerVolumeSpecName "kube-api-access-5rv4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:17.123905 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.123867 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e586cc2-705e-44fb-abaa-dd8993da8faf-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.123905 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.123910 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rv4r\" (UniqueName: \"kubernetes.io/projected/2e586cc2-705e-44fb-abaa-dd8993da8faf-kube-api-access-5rv4r\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.124105 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.123925 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e586cc2-705e-44fb-abaa-dd8993da8faf-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.668210 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.668173 4433 generic.go:356] "Generic (PLEG): container finished" podID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerID="af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651" exitCode=0 Oct 30 00:39:17.668714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.668218 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" Oct 30 00:39:17.668714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.668255 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" event={"ID":"2e586cc2-705e-44fb-abaa-dd8993da8faf","Type":"ContainerDied","Data":"af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651"} Oct 30 00:39:17.668714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.668301 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8" event={"ID":"2e586cc2-705e-44fb-abaa-dd8993da8faf","Type":"ContainerDied","Data":"9b51fff05f3f04432dd05c123e9d2dd0b9a51148ab9b9341723e395f3ed1e138"} Oct 30 00:39:17.668714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.668318 4433 scope.go:122] "RemoveContainer" containerID="af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651" Oct 30 00:39:17.677402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.677380 4433 scope.go:122] "RemoveContainer" containerID="af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651" Oct 30 00:39:17.677688 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:17.677669 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651\": container with ID starting with af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651 not found: ID does not exist" containerID="af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651" Oct 30 00:39:17.677744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.677697 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651"} err="failed to get container status \"af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651\": rpc error: code = NotFound desc = could not find container \"af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651\": container with ID starting with af9c898827d19c335bd7ce4070fbeac1cff3424e12658514b9d77042a854d651 not found: ID does not exist" Oct 30 00:39:17.722545 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.722514 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8"] Oct 30 00:39:17.736005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.735952 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/distinct-2-w0-7505bd-kserve-cc744f9f5-kj7l8"] Oct 30 00:39:17.795848 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:17.795812 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:19.178891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:19.178848 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" path="/var/lib/kubelet/pods/2e586cc2-705e-44fb-abaa-dd8993da8faf/volumes" Oct 30 00:39:19.376455 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:19.376420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:20.359967 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.359921 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" containerID="cri-o://fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f" gracePeriod=45 Oct 30 00:39:20.378666 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.378636 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" containerID="cri-o://a91bd721b0a879abf6d31b70e2c49fad988c636c6abd39484f88591a950bc57e" gracePeriod=45 Oct 30 00:39:20.676950 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.676919 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:39:20.690736 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.690706 4433 generic.go:356] "Generic (PLEG): container finished" podID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerID="fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f" exitCode=0 Oct 30 00:39:20.690948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.690853 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" Oct 30 00:39:20.691052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.691029 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" event={"ID":"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9","Type":"ContainerDied","Data":"fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f"} Oct 30 00:39:20.691102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.691074 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw" event={"ID":"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9","Type":"ContainerDied","Data":"8ccfd0f2a99cc21f3b9fbfb939e8549cc537e5236f6ce2600ed875be7b6e0658"} Oct 30 00:39:20.691102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.691098 4433 scope.go:122] "RemoveContainer" containerID="fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f" Oct 30 00:39:20.692871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.692847 4433 generic.go:356] "Generic (PLEG): container finished" podID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerID="a91bd721b0a879abf6d31b70e2c49fad988c636c6abd39484f88591a950bc57e" exitCode=0 Oct 30 00:39:20.693008 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.692948 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" event={"ID":"a4deb7d9-4a19-418f-a8c3-9d3064257bfd","Type":"ContainerDied","Data":"a91bd721b0a879abf6d31b70e2c49fad988c636c6abd39484f88591a950bc57e"} Oct 30 00:39:20.712662 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.712424 4433 scope.go:122] "RemoveContainer" containerID="fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f" Oct 30 00:39:20.717720 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:20.716851 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f\": container with ID starting with fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f not found: ID does not exist" containerID="fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f" Oct 30 00:39:20.717720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.716894 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f"} err="failed to get container status \"fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f\": rpc error: code = NotFound desc = could not find container \"fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f\": container with ID starting with fd72bb728c52ebcc64f3c4bfe3cab27435c153fffc7b91b20866d06e605d0c4f not found: ID does not exist" Oct 30 00:39:20.728067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.722451 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:39:20.778187 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778143 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-model-cache\") pod \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " Oct 30 00:39:20.778362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778204 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-model-cache\") pod \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " Oct 30 00:39:20.778362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778236 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tmp-dir\") pod \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " Oct 30 00:39:20.778362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778277 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-dshm\") pod \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " Oct 30 00:39:20.778362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778306 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-kube-api-access-htlgx\" (UniqueName: \"kubernetes.io/projected/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-kube-api-access-htlgx\") pod \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " Oct 30 00:39:20.778362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778352 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-dshm\") pod \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " Oct 30 00:39:20.778702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778387 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-home\") pod \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " Oct 30 00:39:20.778702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778425 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tls-certs\") pod \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " Oct 30 00:39:20.778702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778490 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-model-cache" pod "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" (UID: "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.778702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778522 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-home\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-home\") pod \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " Oct 30 00:39:20.778928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778498 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-model-cache" pod "a4deb7d9-4a19-418f-a8c3-9d3064257bfd" (UID: "a4deb7d9-4a19-418f-a8c3-9d3064257bfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.778928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778523 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tmp-dir" pod "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" (UID: "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.778928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778759 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-home" pod "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" (UID: "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.778928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778795 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-home" pod "a4deb7d9-4a19-418f-a8c3-9d3064257bfd" (UID: "a4deb7d9-4a19-418f-a8c3-9d3064257bfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.778928 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778897 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-kube-api-access-w7tjp\" (UniqueName: \"kubernetes.io/projected/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-kube-api-access-w7tjp\") pod \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\" (UID: \"2ffa82da-a0b6-473b-9556-a11cd7e0a0b9\") " Oct 30 00:39:20.779194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.778943 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tls-certs\") pod \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " Oct 30 00:39:20.779194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.779007 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tmp-dir\") pod \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\" (UID: \"a4deb7d9-4a19-418f-a8c3-9d3064257bfd\") " Oct 30 00:39:20.780054 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.780020 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tmp-dir" pod "a4deb7d9-4a19-418f-a8c3-9d3064257bfd" (UID: "a4deb7d9-4a19-418f-a8c3-9d3064257bfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.780398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.780369 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.780398 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.780396 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.780780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.780435 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.780780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.780447 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.780780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.780460 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.780780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.780472 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.781922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.781895 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-kube-api-access-w7tjp" pod "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" (UID: "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9"). InnerVolumeSpecName "kube-api-access-w7tjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:20.782329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.781944 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-dshm" pod "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" (UID: "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.782543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.782510 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-dshm" pod "a4deb7d9-4a19-418f-a8c3-9d3064257bfd" (UID: "a4deb7d9-4a19-418f-a8c3-9d3064257bfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:20.782543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.782519 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tls-certs" pod "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" (UID: "2ffa82da-a0b6-473b-9556-a11cd7e0a0b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:20.782859 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.782837 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tls-certs" pod "a4deb7d9-4a19-418f-a8c3-9d3064257bfd" (UID: "a4deb7d9-4a19-418f-a8c3-9d3064257bfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:20.784684 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.784558 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-kube-api-access-htlgx" pod "a4deb7d9-4a19-418f-a8c3-9d3064257bfd" (UID: "a4deb7d9-4a19-418f-a8c3-9d3064257bfd"). InnerVolumeSpecName "kube-api-access-htlgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:20.881491 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.881448 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.881491 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.881486 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlgx\" (UniqueName: \"kubernetes.io/projected/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-kube-api-access-htlgx\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.881491 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.881497 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.881791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.881507 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.881791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.881516 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7tjp\" (UniqueName: \"kubernetes.io/projected/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9-kube-api-access-w7tjp\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.881791 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.881524 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4deb7d9-4a19-418f-a8c3-9d3064257bfd-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:20.998009 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:20.997922 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:21.062609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.062441 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw"] Oct 30 00:39:21.062609 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.062509 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/distinct-w2-322d20-kserve-c98d94958-58blw"] Oct 30 00:39:21.180388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.180353 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" path="/var/lib/kubelet/pods/2ffa82da-a0b6-473b-9556-a11cd7e0a0b9/volumes" Oct 30 00:39:21.701364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.701282 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" Oct 30 00:39:21.701364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.701307 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx" event={"ID":"a4deb7d9-4a19-418f-a8c3-9d3064257bfd","Type":"ContainerDied","Data":"001ce20f153953e3e5340f690fe5fefc372802be3894966e765bc434e97e4f1b"} Oct 30 00:39:21.701364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.701361 4433 scope.go:122] "RemoveContainer" containerID="a91bd721b0a879abf6d31b70e2c49fad988c636c6abd39484f88591a950bc57e" Oct 30 00:39:21.755543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.755502 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx"] Oct 30 00:39:21.768466 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:21.768435 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-322d20/simulator-w2-322d20-kserve-799c4dccf4-52whx"] Oct 30 00:39:22.267487 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.267453 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942"] Oct 30 00:39:22.267893 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.267711 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" podUID="e7f89b3b-8a1f-4b55-bff9-c22231711f78" containerName="maas-api" containerID="cri-o://4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f" gracePeriod=30 Oct 30 00:39:22.332991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.332623 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:22.687598 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.683816 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:39:22.709422 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.709388 4433 generic.go:356] "Generic (PLEG): container finished" podID="e7f89b3b-8a1f-4b55-bff9-c22231711f78" containerID="4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f" exitCode=0 Oct 30 00:39:22.709944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.709461 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" event={"ID":"e7f89b3b-8a1f-4b55-bff9-c22231711f78","Type":"ContainerDied","Data":"4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f"} Oct 30 00:39:22.709944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.709501 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" event={"ID":"e7f89b3b-8a1f-4b55-bff9-c22231711f78","Type":"ContainerDied","Data":"1b7b5a2befe2895de2ea03f5f9025a1ade16dca82a8fbbfe01455167370a344d"} Oct 30 00:39:22.709944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.709523 4433 scope.go:122] "RemoveContainer" containerID="4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f" Oct 30 00:39:22.709944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.709626 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942" Oct 30 00:39:22.716178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.715536 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-maas-api-tls\") pod \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " Oct 30 00:39:22.716178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.715711 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f89b3b-8a1f-4b55-bff9-c22231711f78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f89b3b-8a1f-4b55-bff9-c22231711f78-openshift-service-ca\") pod \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " Oct 30 00:39:22.716178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.715822 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f89b3b-8a1f-4b55-bff9-c22231711f78-kube-api-access-8w92x\" (UniqueName: \"kubernetes.io/projected/e7f89b3b-8a1f-4b55-bff9-c22231711f78-kube-api-access-8w92x\") pod \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " Oct 30 00:39:22.716178 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.715871 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-metrics-certs\") pod \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\" (UID: \"e7f89b3b-8a1f-4b55-bff9-c22231711f78\") " Oct 30 00:39:22.716891 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.716867 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f89b3b-8a1f-4b55-bff9-c22231711f78-openshift-service-ca" pod "e7f89b3b-8a1f-4b55-bff9-c22231711f78" (UID: "e7f89b3b-8a1f-4b55-bff9-c22231711f78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:22.718773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.718747 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f89b3b-8a1f-4b55-bff9-c22231711f78-kube-api-access-8w92x" pod "e7f89b3b-8a1f-4b55-bff9-c22231711f78" (UID: "e7f89b3b-8a1f-4b55-bff9-c22231711f78"). InnerVolumeSpecName "kube-api-access-8w92x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:22.718773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.718756 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-metrics-certs" pod "e7f89b3b-8a1f-4b55-bff9-c22231711f78" (UID: "e7f89b3b-8a1f-4b55-bff9-c22231711f78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:22.719535 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.719513 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-maas-api-tls" pod "e7f89b3b-8a1f-4b55-bff9-c22231711f78" (UID: "e7f89b3b-8a1f-4b55-bff9-c22231711f78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:22.725623 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.725512 4433 scope.go:122] "RemoveContainer" containerID="4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f" Oct 30 00:39:22.726307 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:22.726272 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f\": container with ID starting with 4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f not found: ID does not exist" containerID="4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f" Oct 30 00:39:22.726467 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.726324 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f"} err="failed to get container status \"4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f\": rpc error: code = NotFound desc = could not find container \"4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f\": container with ID starting with 4902a2f4fd14c6260a2f051ddf1e7e2694d09f7d8c6bbe8992c41479f96e525f not found: ID does not exist" Oct 30 00:39:22.817438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.817408 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f89b3b-8a1f-4b55-bff9-c22231711f78-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:22.817438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.817433 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w92x\" (UniqueName: \"kubernetes.io/projected/e7f89b3b-8a1f-4b55-bff9-c22231711f78-kube-api-access-8w92x\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:22.817438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.817442 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:22.817696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.817453 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f89b3b-8a1f-4b55-bff9-c22231711f78-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:22.879239 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.879141 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:22.988056 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:22.987992 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:23.069818 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:23.069765 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942"] Oct 30 00:39:23.070437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:23.070402 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7505bd-647f5d7f87-lm942"] Oct 30 00:39:23.098673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:23.098640 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:23.179519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:23.179444 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" path="/var/lib/kubelet/pods/a4deb7d9-4a19-418f-a8c3-9d3064257bfd/volumes" Oct 30 00:39:23.179947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:23.179932 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f89b3b-8a1f-4b55-bff9-c22231711f78" path="/var/lib/kubelet/pods/e7f89b3b-8a1f-4b55-bff9-c22231711f78/volumes" Oct 30 00:39:23.528061 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:23.527968 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:24.697062 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:24.697017 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:26.446855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:26.446820 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:27.593045 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:27.593013 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:28.306528 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:28.306488 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:29.605834 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:29.605800 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp"] Oct 30 00:39:29.606284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:29.606090 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" podUID="c9ad66e9-958f-4411-9c86-542cb7739541" containerName="maas-api" containerID="cri-o://ebe1a154dff80bf3f7342a7b0bab9dc5dc0364eb60b54b8c6209b3cc90bfb168" gracePeriod=30 Oct 30 00:39:29.672664 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:29.672630 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:29.774024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:29.773992 4433 generic.go:356] "Generic (PLEG): container finished" podID="c9ad66e9-958f-4411-9c86-542cb7739541" containerID="ebe1a154dff80bf3f7342a7b0bab9dc5dc0364eb60b54b8c6209b3cc90bfb168" exitCode=0 Oct 30 00:39:29.774186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:29.774077 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" event={"ID":"c9ad66e9-958f-4411-9c86-542cb7739541","Type":"ContainerDied","Data":"ebe1a154dff80bf3f7342a7b0bab9dc5dc0364eb60b54b8c6209b3cc90bfb168"} Oct 30 00:39:29.923527 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:29.923496 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:39:30.060997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.060961 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-metrics-certs\") pod \"c9ad66e9-958f-4411-9c86-542cb7739541\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " Oct 30 00:39:30.061207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.061032 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-maas-api-tls\") pod \"c9ad66e9-958f-4411-9c86-542cb7739541\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " Oct 30 00:39:30.061207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.061076 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ad66e9-958f-4411-9c86-542cb7739541-kube-api-access-cgtjl\" (UniqueName: \"kubernetes.io/projected/c9ad66e9-958f-4411-9c86-542cb7739541-kube-api-access-cgtjl\") pod \"c9ad66e9-958f-4411-9c86-542cb7739541\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " Oct 30 00:39:30.061207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.061159 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ad66e9-958f-4411-9c86-542cb7739541-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad66e9-958f-4411-9c86-542cb7739541-openshift-service-ca\") pod \"c9ad66e9-958f-4411-9c86-542cb7739541\" (UID: \"c9ad66e9-958f-4411-9c86-542cb7739541\") " Oct 30 00:39:30.061673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.061639 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ad66e9-958f-4411-9c86-542cb7739541-openshift-service-ca" pod "c9ad66e9-958f-4411-9c86-542cb7739541" (UID: "c9ad66e9-958f-4411-9c86-542cb7739541"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:30.062023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.061807 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ad66e9-958f-4411-9c86-542cb7739541-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:30.063377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.063349 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-metrics-certs" pod "c9ad66e9-958f-4411-9c86-542cb7739541" (UID: "c9ad66e9-958f-4411-9c86-542cb7739541"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:30.063708 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.063681 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-maas-api-tls" pod "c9ad66e9-958f-4411-9c86-542cb7739541" (UID: "c9ad66e9-958f-4411-9c86-542cb7739541"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:30.063831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.063681 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ad66e9-958f-4411-9c86-542cb7739541-kube-api-access-cgtjl" pod "c9ad66e9-958f-4411-9c86-542cb7739541" (UID: "c9ad66e9-958f-4411-9c86-542cb7739541"). InnerVolumeSpecName "kube-api-access-cgtjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:30.163548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.163452 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:30.163548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.163489 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ad66e9-958f-4411-9c86-542cb7739541-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:30.163548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.163504 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtjl\" (UniqueName: \"kubernetes.io/projected/c9ad66e9-958f-4411-9c86-542cb7739541-kube-api-access-cgtjl\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:30.183174 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.183013 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.293503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.293471 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.404079 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.404044 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.540092 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.539997 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.646173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.646137 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.780831 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.780798 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" Oct 30 00:39:30.781003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.780829 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp" event={"ID":"c9ad66e9-958f-4411-9c86-542cb7739541","Type":"ContainerDied","Data":"904e78be195e00a2cc9bae890d6310312da661ad4557e93e596789b08364a74b"} Oct 30 00:39:30.781003 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.780887 4433 scope.go:122] "RemoveContainer" containerID="ebe1a154dff80bf3f7342a7b0bab9dc5dc0364eb60b54b8c6209b3cc90bfb168" Oct 30 00:39:30.825369 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.825321 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.850948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.850918 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp"] Oct 30 00:39:30.871855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.871714 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-322d20-5c7ddd7457-vsjrp"] Oct 30 00:39:30.898176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.898131 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr"] Oct 30 00:39:30.898367 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.898208 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n"] Oct 30 00:39:30.898461 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.898433 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" containerID="cri-o://ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4" gracePeriod=30 Oct 30 00:39:30.898744 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.898667 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" containerID="cri-o://284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c" gracePeriod=30 Oct 30 00:39:30.901617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.901558 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs"] Oct 30 00:39:30.901973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.901813 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" containerID="cri-o://4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4" gracePeriod=30 Oct 30 00:39:30.943833 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:30.942870 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:31.178696 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:31.178666 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ad66e9-958f-4411-9c86-542cb7739541" path="/var/lib/kubelet/pods/c9ad66e9-958f-4411-9c86-542cb7739541/volumes" Oct 30 00:39:32.085214 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:32.085181 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:33.681514 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:33.681475 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:34.191600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:34.191528 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:39:34.191789 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:34.191609 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:39:35.184193 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:35.184156 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:36.171043 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.170976 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:39:36.206452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.205391 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:39:36.301615 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.299765 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:39:36.347036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347002 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-socket\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.347248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347052 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istiod-ca-cert\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.347248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347082 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-data\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.347248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347124 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-credential-socket\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.347248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347161 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-data\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.347248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347191 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-kube-api-access-dw4s5\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-kube-api-access-dw4s5\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.347248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347230 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-envoy\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.347548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347372 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-socket" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.347548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347429 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-token\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-token\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.347548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347481 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-istio-token\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-istio-token\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.347548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347526 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-socket\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.347548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347508 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-data" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347552 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-data" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347539 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-credential-socket" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347601 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istiod-ca-cert" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347561 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-credential-socket\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347765 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-envoy\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347810 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-credential-socket" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347822 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-socket" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347862 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3993addf-2a45-44a7-9714-ad89b90ea081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3993addf-2a45-44a7-9714-ad89b90ea081-istiod-ca-cert\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.347937 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-podinfo\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348005 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-certs\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348039 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-kube-api-access-hjrmz\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-kube-api-access-hjrmz\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348113 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-certs\") pod \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\" (UID: \"d8fb91fd-aa1a-43a5-a388-24ba8562fa23\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348159 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3993addf-2a45-44a7-9714-ad89b90ea081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3993addf-2a45-44a7-9714-ad89b90ea081-istio-podinfo\") pod \"3993addf-2a45-44a7-9714-ad89b90ea081\" (UID: \"3993addf-2a45-44a7-9714-ad89b90ea081\") " Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348204 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3993addf-2a45-44a7-9714-ad89b90ea081-istiod-ca-cert" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348382 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-certs" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348741 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348764 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348780 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348794 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348808 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348822 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348836 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348850 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3993addf-2a45-44a7-9714-ad89b90ea081-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.348863 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.349049 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-certs" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.351264 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3993addf-2a45-44a7-9714-ad89b90ea081-istio-podinfo" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.351465 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-podinfo" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:36.353878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.353311 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-kube-api-access-hjrmz" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "kube-api-access-hjrmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.355842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.355814 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-envoy" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.355842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.355823 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-token" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.355842 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.355841 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-istio-token" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.356128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.355965 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-kube-api-access-dw4s5" pod "3993addf-2a45-44a7-9714-ad89b90ea081" (UID: "3993addf-2a45-44a7-9714-ad89b90ea081"). InnerVolumeSpecName "kube-api-access-dw4s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.357088 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.357063 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-envoy" pod "d8fb91fd-aa1a-43a5-a388-24ba8562fa23" (UID: "d8fb91fd-aa1a-43a5-a388-24ba8562fa23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.450364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450266 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-envoy\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450350 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-data\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450390 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-socket\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450418 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/509d7476-b7d1-4657-b779-a2876047d78c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509d7476-b7d1-4657-b779-a2876047d78c-istio-podinfo\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450477 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-istio-token\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-istio-token\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450555 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-credential-socket\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450631 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450623 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-certs\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450696 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/509d7476-b7d1-4657-b779-a2876047d78c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509d7476-b7d1-4657-b779-a2876047d78c-istiod-ca-cert\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.450973 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.450737 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-kube-api-access-drzgm\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-kube-api-access-drzgm\") pod \"509d7476-b7d1-4657-b779-a2876047d78c\" (UID: \"509d7476-b7d1-4657-b779-a2876047d78c\") " Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451254 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451285 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451299 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451313 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451327 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjrmz\" (UniqueName: \"kubernetes.io/projected/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-kube-api-access-hjrmz\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451342 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fb91fd-aa1a-43a5-a388-24ba8562fa23-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451356 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3993addf-2a45-44a7-9714-ad89b90ea081-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451369 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw4s5\" (UniqueName: \"kubernetes.io/projected/3993addf-2a45-44a7-9714-ad89b90ea081-kube-api-access-dw4s5\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.451405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451384 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3993addf-2a45-44a7-9714-ad89b90ea081-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.452166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.451886 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-certs" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.452166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.452090 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-data" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.452166 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.452159 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-socket" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.452629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.452299 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509d7476-b7d1-4657-b779-a2876047d78c-istiod-ca-cert" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:36.452629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.452530 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-credential-socket" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.454039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.454015 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-kube-api-access-drzgm" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "kube-api-access-drzgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.455183 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.455155 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-istio-token" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.456138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.456113 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-envoy" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.456329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.456312 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/509d7476-b7d1-4657-b779-a2876047d78c-istio-podinfo" pod "509d7476-b7d1-4657-b779-a2876047d78c" (UID: "509d7476-b7d1-4657-b779-a2876047d78c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:36.552244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552202 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552244 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552241 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509d7476-b7d1-4657-b779-a2876047d78c-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552254 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drzgm\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-kube-api-access-drzgm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552268 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552282 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552293 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552305 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509d7476-b7d1-4657-b779-a2876047d78c-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552316 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509d7476-b7d1-4657-b779-a2876047d78c-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.552483 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.552327 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509d7476-b7d1-4657-b779-a2876047d78c-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.798714 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.798626 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:36.820958 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.820919 4433 generic.go:356] "Generic (PLEG): container finished" podID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerID="284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c" exitCode=0 Oct 30 00:39:36.821143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.820968 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" Oct 30 00:39:36.821143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.821004 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" event={"ID":"d8fb91fd-aa1a-43a5-a388-24ba8562fa23","Type":"ContainerDied","Data":"284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c"} Oct 30 00:39:36.821143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.821059 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr" event={"ID":"d8fb91fd-aa1a-43a5-a388-24ba8562fa23","Type":"ContainerDied","Data":"cdd0aa6b8011679c5679eaacbf4281076b852517a3f417ef41915e27636df98f"} Oct 30 00:39:36.821143 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.821082 4433 scope.go:122] "RemoveContainer" containerID="284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c" Oct 30 00:39:36.822517 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.822492 4433 generic.go:356] "Generic (PLEG): container finished" podID="3993addf-2a45-44a7-9714-ad89b90ea081" containerID="ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4" exitCode=0 Oct 30 00:39:36.822647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.822539 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" Oct 30 00:39:36.822647 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.822539 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" event={"ID":"3993addf-2a45-44a7-9714-ad89b90ea081","Type":"ContainerDied","Data":"ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4"} Oct 30 00:39:36.822788 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.822653 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n" event={"ID":"3993addf-2a45-44a7-9714-ad89b90ea081","Type":"ContainerDied","Data":"c3bbb16915811ce488180c833cade5f4eae68f5fa540651baf7bb16c96718bb4"} Oct 30 00:39:36.824550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.824526 4433 generic.go:356] "Generic (PLEG): container finished" podID="509d7476-b7d1-4657-b779-a2876047d78c" containerID="4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4" exitCode=0 Oct 30 00:39:36.824681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.824565 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" Oct 30 00:39:36.824681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.824565 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" event={"ID":"509d7476-b7d1-4657-b779-a2876047d78c","Type":"ContainerDied","Data":"4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4"} Oct 30 00:39:36.824681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.824640 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs" event={"ID":"509d7476-b7d1-4657-b779-a2876047d78c","Type":"ContainerDied","Data":"4ddff60f197056506b589fd21dfe389ce22ba2c7e262cbfae826595267334060"} Oct 30 00:39:36.831729 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.831712 4433 scope.go:122] "RemoveContainer" containerID="284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c" Oct 30 00:39:36.831982 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:36.831954 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c\": container with ID starting with 284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c not found: ID does not exist" containerID="284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c" Oct 30 00:39:36.832066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.831982 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c"} err="failed to get container status \"284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c\": rpc error: code = NotFound desc = could not find container \"284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c\": container with ID starting with 284b31f4e34a48170c3a22fefb9f57638f4798723eeff705c0646c2ce933ab1c not found: ID does not exist" Oct 30 00:39:36.832066 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.832006 4433 scope.go:122] "RemoveContainer" containerID="ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4" Oct 30 00:39:36.861519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.841384 4433 scope.go:122] "RemoveContainer" containerID="ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4" Oct 30 00:39:36.861519 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:36.844835 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4\": container with ID starting with ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4 not found: ID does not exist" containerID="ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4" Oct 30 00:39:36.861519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.844878 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4"} err="failed to get container status \"ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4\": rpc error: code = NotFound desc = could not find container \"ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4\": container with ID starting with ff99f8858d50f98f772ddac0550022f0cd99091c1ba668969ed168323584baa4 not found: ID does not exist" Oct 30 00:39:36.861519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.844905 4433 scope.go:122] "RemoveContainer" containerID="4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4" Oct 30 00:39:36.870158 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.870137 4433 scope.go:122] "RemoveContainer" containerID="77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953" Oct 30 00:39:36.878954 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.878934 4433 scope.go:122] "RemoveContainer" containerID="4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4" Oct 30 00:39:36.879236 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:36.879218 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4\": container with ID starting with 4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4 not found: ID does not exist" containerID="4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4" Oct 30 00:39:36.879285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.879248 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4"} err="failed to get container status \"4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4\": rpc error: code = NotFound desc = could not find container \"4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4\": container with ID starting with 4cbc7b1bf0f9891f7611e8f7d825e14acd8cae770499c65b7957564d603ab6c4 not found: ID does not exist" Oct 30 00:39:36.879285 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.879266 4433 scope.go:122] "RemoveContainer" containerID="77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953" Oct 30 00:39:36.879505 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:36.879489 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953\": container with ID starting with 77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953 not found: ID does not exist" containerID="77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953" Oct 30 00:39:36.879546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.879510 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953"} err="failed to get container status \"77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953\": rpc error: code = NotFound desc = could not find container \"77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953\": container with ID starting with 77d61273991e14639fba770d1e3f8b859cf12e529d816dd4d65ebba134bd6953 not found: ID does not exist" Oct 30 00:39:36.884852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.884829 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n"] Oct 30 00:39:36.888218 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.888197 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-qmg2n"] Oct 30 00:39:36.923795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.923768 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs"] Oct 30 00:39:36.926070 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.926046 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-q8sxs"] Oct 30 00:39:36.960944 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.960917 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr"] Oct 30 00:39:36.964772 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:36.964748 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-jwbhr"] Oct 30 00:39:37.056701 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.056615 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:37.098380 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:37.098345 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:39:37.166837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.166806 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:37.187743 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.187710 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" path="/var/lib/kubelet/pods/3993addf-2a45-44a7-9714-ad89b90ea081/volumes" Oct 30 00:39:37.188186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.188172 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="509d7476-b7d1-4657-b779-a2876047d78c" path="/var/lib/kubelet/pods/509d7476-b7d1-4657-b779-a2876047d78c/volumes" Oct 30 00:39:37.188681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.188666 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" path="/var/lib/kubelet/pods/d8fb91fd-aa1a-43a5-a388-24ba8562fa23/volumes" Oct 30 00:39:37.405821 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.405786 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh"] Oct 30 00:39:37.406262 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.406065 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" containerID="cri-o://3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49" gracePeriod=30 Oct 30 00:39:37.409232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.409200 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr"] Oct 30 00:39:37.409458 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.409420 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" containerID="cri-o://192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6" gracePeriod=30 Oct 30 00:39:37.420153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.419343 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g"] Oct 30 00:39:37.420153 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.419547 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" containerID="cri-o://f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54" gracePeriod=30 Oct 30 00:39:37.454237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.454187 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:37.608377 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:37.608343 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:38.235648 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:38.235606 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:38.391731 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:38.391666 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:38.525868 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:38.525730 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:38.894132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:38.894096 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:39.666005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:39.665972 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:39.852816 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:39.852779 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:40.181429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:40.181379 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:40.322948 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:40.322914 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:40.448186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:40.448099 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:40.554421 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:40.554390 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:40.666995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:40.666958 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:42.258230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.258190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:42.719230 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.719172 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:39:42.730162 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.730127 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:42.756752 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.756722 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:39:42.761490 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.761462 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:39:42.822746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.822698 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-credential-socket\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.822746 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.822753 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-token\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-token\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.823044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.822789 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-podinfo\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.823044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.822831 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-certs\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.823044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.822898 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-socket\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.823044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.822954 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-credential-socket\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.823044 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.822992 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-socket\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.823323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823047 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istiod-ca-cert\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.823323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823111 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34827380-65c5-479c-bf82-35dfbd7bc003-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34827380-65c5-479c-bf82-35dfbd7bc003-istiod-ca-cert\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.823323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823213 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-kube-api-access-9swg2\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-kube-api-access-9swg2\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.823323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823266 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-token\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-token\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.823323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823296 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d44d38e-54b0-473f-8f81-64328091c5f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d44d38e-54b0-473f-8f81-64328091c5f7-istiod-ca-cert\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823346 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-kube-api-access-tslgw\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-kube-api-access-tslgw\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823386 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-envoy\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823439 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-podinfo\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823486 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34827380-65c5-479c-bf82-35dfbd7bc003-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34827380-65c5-479c-bf82-35dfbd7bc003-istio-podinfo\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823524 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-envoy\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823565 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-istio-token\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-istio-token\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823648 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-kube-api-access-fdp9v\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-kube-api-access-fdp9v\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823691 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-certs\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.823753 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823740 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-data\") pod \"34827380-65c5-479c-bf82-35dfbd7bc003\" (UID: \"34827380-65c5-479c-bf82-35dfbd7bc003\") " Oct 30 00:39:42.824258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823775 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-data\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.824258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823826 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-envoy\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.824258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823879 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-data\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.824258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823953 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-socket\") pod \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\" (UID: \"ade4e4dc-fc17-4e27-901f-9321d6ea55e4\") " Oct 30 00:39:42.824258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.823996 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-certs\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.824258 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.824041 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-credential-socket\") pod \"2d44d38e-54b0-473f-8f81-64328091c5f7\" (UID: \"2d44d38e-54b0-473f-8f81-64328091c5f7\") " Oct 30 00:39:42.824830 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.824802 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-credential-socket" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.827378 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.825037 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-credential-socket" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.830685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.829961 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-data" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.830685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.830234 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-socket" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.830685 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.830433 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-certs" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.831602 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.831374 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-certs" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.832470 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.832214 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istiod-ca-cert" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:42.845651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.835106 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34827380-65c5-479c-bf82-35dfbd7bc003-istiod-ca-cert" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:42.845651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.837703 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-data" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.845651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.840644 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-socket" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.845651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.845241 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-kube-api-access-fdp9v" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "kube-api-access-fdp9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:42.845651 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.845494 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-credential-socket" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.846811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.845740 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-certs" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.846811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.846317 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-socket" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.846811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.846362 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d44d38e-54b0-473f-8f81-64328091c5f7-istiod-ca-cert" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:42.848929 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.848783 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-data" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.849878 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.849795 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:42.850381 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.850356 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-podinfo" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:42.852775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.852697 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-podinfo" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:42.852775 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.852706 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-kube-api-access-9swg2" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "kube-api-access-9swg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:42.855393 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.855350 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34827380-65c5-479c-bf82-35dfbd7bc003-istio-podinfo" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:42.859429 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.859201 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-kube-api-access-tslgw" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "kube-api-access-tslgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:42.859934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.859906 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-istio-token" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:42.860155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.859934 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-token" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:42.860155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.860096 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-envoy" pod "34827380-65c5-479c-bf82-35dfbd7bc003" (UID: "34827380-65c5-479c-bf82-35dfbd7bc003"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.860155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.860121 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-token" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:42.860500 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.860463 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-envoy" pod "ade4e4dc-fc17-4e27-901f-9321d6ea55e4" (UID: "ade4e4dc-fc17-4e27-901f-9321d6ea55e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.864370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.864339 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-envoy" pod "2d44d38e-54b0-473f-8f81-64328091c5f7" (UID: "2d44d38e-54b0-473f-8f81-64328091c5f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:42.866645 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.866617 4433 generic.go:356] "Generic (PLEG): container finished" podID="34827380-65c5-479c-bf82-35dfbd7bc003" containerID="192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6" exitCode=0 Oct 30 00:39:42.866760 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.866747 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" Oct 30 00:39:42.866870 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.866845 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" event={"ID":"34827380-65c5-479c-bf82-35dfbd7bc003","Type":"ContainerDied","Data":"192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6"} Oct 30 00:39:42.867188 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.867165 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr" event={"ID":"34827380-65c5-479c-bf82-35dfbd7bc003","Type":"ContainerDied","Data":"6646511561df5e34514e4499ff5581d7cb00a090bfd1853884f29a2777a5259d"} Oct 30 00:39:42.867240 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.867216 4433 scope.go:122] "RemoveContainer" containerID="192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6" Oct 30 00:39:42.870300 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.870279 4433 generic.go:356] "Generic (PLEG): container finished" podID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerID="3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49" exitCode=0 Oct 30 00:39:42.870391 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.870380 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" event={"ID":"2d44d38e-54b0-473f-8f81-64328091c5f7","Type":"ContainerDied","Data":"3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49"} Oct 30 00:39:42.870454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.870425 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" event={"ID":"2d44d38e-54b0-473f-8f81-64328091c5f7","Type":"ContainerDied","Data":"335e4f94aec20e33dd09bca86b9c34fb4ecb0da062710994323e6c8af84325ff"} Oct 30 00:39:42.870519 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.870503 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh" Oct 30 00:39:42.879432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.877930 4433 generic.go:356] "Generic (PLEG): container finished" podID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerID="f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54" exitCode=0 Oct 30 00:39:42.879432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.877979 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" event={"ID":"ade4e4dc-fc17-4e27-901f-9321d6ea55e4","Type":"ContainerDied","Data":"f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54"} Oct 30 00:39:42.879432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.878021 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" event={"ID":"ade4e4dc-fc17-4e27-901f-9321d6ea55e4","Type":"ContainerDied","Data":"5287c7c4de876762efef4527d6d53bbd9e40a4379775ba2375983221cdef352e"} Oct 30 00:39:42.879432 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.878081 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g" Oct 30 00:39:42.882591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.882557 4433 scope.go:122] "RemoveContainer" containerID="6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee" Oct 30 00:39:42.892955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.892897 4433 scope.go:122] "RemoveContainer" containerID="192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6" Oct 30 00:39:42.893547 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:42.893523 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6\": container with ID starting with 192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6 not found: ID does not exist" containerID="192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6" Oct 30 00:39:42.893672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.893557 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6"} err="failed to get container status \"192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6\": rpc error: code = NotFound desc = could not find container \"192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6\": container with ID starting with 192aa5a701e2329a31109550d797b34b34191f68462c4f7694df87773163c8e6 not found: ID does not exist" Oct 30 00:39:42.893672 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.893620 4433 scope.go:122] "RemoveContainer" containerID="6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee" Oct 30 00:39:42.893923 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:42.893896 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee\": container with ID starting with 6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee not found: ID does not exist" containerID="6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee" Oct 30 00:39:42.893991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.893931 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee"} err="failed to get container status \"6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee\": rpc error: code = NotFound desc = could not find container \"6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee\": container with ID starting with 6e9b0eecf7dfc49adaaa9d6f648b43f2cf365d04456cea869136184090af33ee not found: ID does not exist" Oct 30 00:39:42.893991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.893951 4433 scope.go:122] "RemoveContainer" containerID="3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49" Oct 30 00:39:42.904460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.904363 4433 scope.go:122] "RemoveContainer" containerID="3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49" Oct 30 00:39:42.904809 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:42.904780 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49\": container with ID starting with 3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49 not found: ID does not exist" containerID="3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49" Oct 30 00:39:42.904923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.904819 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49"} err="failed to get container status \"3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49\": rpc error: code = NotFound desc = could not find container \"3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49\": container with ID starting with 3c0e1afd7a23bce7a38ab1e93b441e7f656c1ebc87e9cf5da5ec43ae992deb49 not found: ID does not exist" Oct 30 00:39:42.904923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.904845 4433 scope.go:122] "RemoveContainer" containerID="f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54" Oct 30 00:39:42.917807 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.917780 4433 scope.go:122] "RemoveContainer" containerID="f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54" Oct 30 00:39:42.918142 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:39:42.918122 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54\": container with ID starting with f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54 not found: ID does not exist" containerID="f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54" Oct 30 00:39:42.918222 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.918150 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54"} err="failed to get container status \"f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54\": rpc error: code = NotFound desc = could not find container \"f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54\": container with ID starting with f03e44ae68a42c0016661dd1365324f1c8f639223275c9438c5eff04f23efd54 not found: ID does not exist" Oct 30 00:39:42.925423 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925400 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925423 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925422 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925435 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925444 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925453 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34827380-65c5-479c-bf82-35dfbd7bc003-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925465 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swg2\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-kube-api-access-9swg2\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925480 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925490 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d44d38e-54b0-473f-8f81-64328091c5f7-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925498 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tslgw\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-kube-api-access-tslgw\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925505 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925513 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925521 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34827380-65c5-479c-bf82-35dfbd7bc003-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925528 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925535 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925548 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdp9v\" (UniqueName: \"kubernetes.io/projected/34827380-65c5-479c-bf82-35dfbd7bc003-kube-api-access-fdp9v\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925559 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925568 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925591 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.925560 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925603 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925612 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925622 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925635 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925643 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d44d38e-54b0-473f-8f81-64328091c5f7-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925650 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34827380-65c5-479c-bf82-35dfbd7bc003-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925659 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d44d38e-54b0-473f-8f81-64328091c5f7-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925666 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.926136 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.925673 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade4e4dc-fc17-4e27-901f-9321d6ea55e4-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:39:42.937960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.937935 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g"] Oct 30 00:39:42.941263 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.941240 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-n6t5g"] Oct 30 00:39:42.977845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.977809 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr"] Oct 30 00:39:42.982064 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:42.982035 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-tmlqr"] Oct 30 00:39:43.017720 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:43.017683 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh"] Oct 30 00:39:43.021299 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:43.021275 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-xwtxh"] Oct 30 00:39:43.179513 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:43.179430 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" path="/var/lib/kubelet/pods/2d44d38e-54b0-473f-8f81-64328091c5f7/volumes" Oct 30 00:39:43.180038 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:43.180016 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" path="/var/lib/kubelet/pods/34827380-65c5-479c-bf82-35dfbd7bc003/volumes" Oct 30 00:39:43.180452 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:43.180438 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" path="/var/lib/kubelet/pods/ade4e4dc-fc17-4e27-901f-9321d6ea55e4/volumes" Oct 30 00:39:43.682100 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:43.682060 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:43.874092 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:43.874060 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:45.531646 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:45.531605 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:46.537709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:46.537672 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:46.653449 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:46.653414 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:46.866559 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:46.866525 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:46.973294 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:46.973253 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:47.100333 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:47.100298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:48.741310 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:48.740045 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:50.294426 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:50.294389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:51.881625 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:51.881564 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:53.377591 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:53.377543 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:54.840237 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:54.840200 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:55.520673 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:55.520633 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:39:55.523504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:55.523476 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wr7jg_079ae719-a70f-4d46-923c-aeb73ce07125/ovn-acl-logging/0.log" Oct 30 00:39:56.371504 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:56.371456 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:57.702185 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:57.702143 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:57.845793 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:57.845490 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:57.959634 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:57.959523 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:58.295886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:58.295799 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:59.156189 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:59.156144 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:59.273810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:59.273766 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:59.378055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:59.378018 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:59.792785 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:59.792748 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:59.922281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:39:59.922246 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:01.401172 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:01.401138 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:02.991270 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:02.991221 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:04.545284 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:04.545250 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:05.716055 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:05.716019 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:05.830255 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:05.830218 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:05.983126 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:05.983044 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:06.045248 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:06.045214 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s"] Oct 30 00:40:06.045523 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:06.045496 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" containerID="cri-o://0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52" gracePeriod=30 Oct 30 00:40:06.095976 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:06.095941 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:07.098806 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:40:07.098765 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:40:07.589770 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:07.589736 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:09.094915 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:09.094875 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:10.686329 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:10.686292 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:11.303537 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.303507 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:40:11.433383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433295 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26be440e-117f-46bd-957c-f59d9616abcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26be440e-117f-46bd-957c-f59d9616abcb-istiod-ca-cert\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433383 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433336 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-kube-api-access-mtfdz\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-kube-api-access-mtfdz\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433386 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-certs\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433478 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-istio-token\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-istio-token\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433544 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26be440e-117f-46bd-957c-f59d9616abcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26be440e-117f-46bd-957c-f59d9616abcb-istio-podinfo\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433681 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433626 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-envoy\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433725 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-certs" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.433912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433753 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-data\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433778 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-socket\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433800 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-credential-socket\") pod \"26be440e-117f-46bd-957c-f59d9616abcb\" (UID: \"26be440e-117f-46bd-957c-f59d9616abcb\") " Oct 30 00:40:11.433912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.433843 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26be440e-117f-46bd-957c-f59d9616abcb-istiod-ca-cert" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:40:11.434186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.434102 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-socket" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.434186 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.434132 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-data" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.434337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.434305 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-credential-socket" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.434409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.434377 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26be440e-117f-46bd-957c-f59d9616abcb-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.434409 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.434398 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.434493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.434410 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.434493 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.434423 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.435832 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.435811 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-kube-api-access-mtfdz" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "kube-api-access-mtfdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:11.435923 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.435845 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-envoy" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.436082 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.436061 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26be440e-117f-46bd-957c-f59d9616abcb-istio-podinfo" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:40:11.436130 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.436093 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-istio-token" pod "26be440e-117f-46bd-957c-f59d9616abcb" (UID: "26be440e-117f-46bd-957c-f59d9616abcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:11.535096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.535059 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.535096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.535090 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfdz\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-kube-api-access-mtfdz\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.535096 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.535100 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26be440e-117f-46bd-957c-f59d9616abcb-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.535324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.535109 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26be440e-117f-46bd-957c-f59d9616abcb-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.535324 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:11.535118 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26be440e-117f-46bd-957c-f59d9616abcb-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:12.039718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.039683 4433 generic.go:356] "Generic (PLEG): container finished" podID="26be440e-117f-46bd-957c-f59d9616abcb" containerID="0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52" exitCode=0 Oct 30 00:40:12.040223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.039730 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" Oct 30 00:40:12.040223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.039743 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" event={"ID":"26be440e-117f-46bd-957c-f59d9616abcb","Type":"ContainerDied","Data":"0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52"} Oct 30 00:40:12.040223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.039847 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s" event={"ID":"26be440e-117f-46bd-957c-f59d9616abcb","Type":"ContainerDied","Data":"606d1cf18dcb3b8803e83958162456b97b5df7aab4eff9219665b0a2ac07adfa"} Oct 30 00:40:12.040223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.039868 4433 scope.go:122] "RemoveContainer" containerID="0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52" Oct 30 00:40:12.049852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.049831 4433 scope.go:122] "RemoveContainer" containerID="0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52" Oct 30 00:40:12.050156 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:40:12.050137 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52\": container with ID starting with 0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52 not found: ID does not exist" containerID="0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52" Oct 30 00:40:12.050217 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.050170 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52"} err="failed to get container status \"0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52\": rpc error: code = NotFound desc = could not find container \"0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52\": container with ID starting with 0b5581c020e900495f7aa883539fcc00b6d3b98c7e415e005eacf55a1206de52 not found: ID does not exist" Oct 30 00:40:12.108587 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.108544 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s"] Oct 30 00:40:12.121772 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.121747 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-kkm2s"] Oct 30 00:40:12.189789 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:12.189758 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:13.178182 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:13.178149 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26be440e-117f-46bd-957c-f59d9616abcb" path="/var/lib/kubelet/pods/26be440e-117f-46bd-957c-f59d9616abcb/volumes" Oct 30 00:40:13.687600 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:13.687552 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:15.274804 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:15.274770 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:16.786914 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:16.786868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:17.244810 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:17.244766 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:17.353665 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:17.353625 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:18.105202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:18.105158 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:18.572843 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:18.572808 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:18.680922 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:18.680891 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:19.231475 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:19.231441 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:20.179503 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:20.179462 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:21.782548 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:21.782503 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:23.302308 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:23.302266 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:24.899986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:24.899946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:25.789526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:25.789492 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:25.897932 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:25.897899 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:26.140252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:26.140213 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:26.394164 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:26.394074 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:26.617128 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:26.617094 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:28.086934 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:28.086901 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:29.688959 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:29.688907 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:31.186484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:31.186446 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:31.576040 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:31.576011 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth"] Oct 30 00:40:31.589133 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:31.589105 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd"] Oct 30 00:40:31.698102 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:31.698059 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:32.773592 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:32.773534 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:34.283337 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:34.283303 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:35.872621 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:35.872557 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:37.098350 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:40:37.098314 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:40:37.400947 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:37.400856 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:37.516876 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:37.516123 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:40.560824 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:40.560790 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:41.147051 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:41.147014 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:46.604895 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:46.604850 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" containerID="cri-o://615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946" gracePeriod=45 Oct 30 00:40:46.612104 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:46.612072 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" containerID="cri-o://e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e" gracePeriod=45 Oct 30 00:40:46.865607 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:46.865518 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:40:46.893298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:46.893272 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:40:47.002260 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002228 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tls-certs\") pod \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " Oct 30 00:40:47.002454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002272 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-tmp-dir\") pod \"af12066d-62f8-49e1-a886-7eb0b1bfae07\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " Oct 30 00:40:47.002454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002318 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-dshm\") pod \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " Oct 30 00:40:47.002454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002352 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tmp-dir\") pod \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " Oct 30 00:40:47.002454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002381 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871d04cb-19f3-4fc4-abff-ee5f3483da6b-kube-api-access-f6qdc\" (UniqueName: \"kubernetes.io/projected/871d04cb-19f3-4fc4-abff-ee5f3483da6b-kube-api-access-f6qdc\") pod \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " Oct 30 00:40:47.002454 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002446 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-model-cache\") pod \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " Oct 30 00:40:47.002792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002516 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-model-cache\") pod \"af12066d-62f8-49e1-a886-7eb0b1bfae07\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " Oct 30 00:40:47.002792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002601 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-tmp-dir" pod "af12066d-62f8-49e1-a886-7eb0b1bfae07" (UID: "af12066d-62f8-49e1-a886-7eb0b1bfae07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.002792 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002707 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tmp-dir" pod "871d04cb-19f3-4fc4-abff-ee5f3483da6b" (UID: "871d04cb-19f3-4fc4-abff-ee5f3483da6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.002960 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.002902 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-model-cache" pod "af12066d-62f8-49e1-a886-7eb0b1bfae07" (UID: "af12066d-62f8-49e1-a886-7eb0b1bfae07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.003146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003116 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af12066d-62f8-49e1-a886-7eb0b1bfae07-kube-api-access-k94xh\" (UniqueName: \"kubernetes.io/projected/af12066d-62f8-49e1-a886-7eb0b1bfae07-kube-api-access-k94xh\") pod \"af12066d-62f8-49e1-a886-7eb0b1bfae07\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " Oct 30 00:40:47.003298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003209 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-model-cache" pod "871d04cb-19f3-4fc4-abff-ee5f3483da6b" (UID: "871d04cb-19f3-4fc4-abff-ee5f3483da6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.003298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003221 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-home\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-home\") pod \"af12066d-62f8-49e1-a886-7eb0b1bfae07\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " Oct 30 00:40:47.003298 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003293 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af12066d-62f8-49e1-a886-7eb0b1bfae07-tls-certs\" (UniqueName: \"kubernetes.io/secret/af12066d-62f8-49e1-a886-7eb0b1bfae07-tls-certs\") pod \"af12066d-62f8-49e1-a886-7eb0b1bfae07\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " Oct 30 00:40:47.003494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003426 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-home\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-home\") pod \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\" (UID: \"871d04cb-19f3-4fc4-abff-ee5f3483da6b\") " Oct 30 00:40:47.003494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003471 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-dshm\") pod \"af12066d-62f8-49e1-a886-7eb0b1bfae07\" (UID: \"af12066d-62f8-49e1-a886-7eb0b1bfae07\") " Oct 30 00:40:47.003494 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003471 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-home" pod "af12066d-62f8-49e1-a886-7eb0b1bfae07" (UID: "af12066d-62f8-49e1-a886-7eb0b1bfae07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.004067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.003976 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.004067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.004005 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.004067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.004019 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-model-cache\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.004067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.004030 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.004067 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.004044 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-tmp-dir\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.004417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.004121 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-home" pod "871d04cb-19f3-4fc4-abff-ee5f3483da6b" (UID: "871d04cb-19f3-4fc4-abff-ee5f3483da6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.005089 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.005066 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871d04cb-19f3-4fc4-abff-ee5f3483da6b-kube-api-access-f6qdc" pod "871d04cb-19f3-4fc4-abff-ee5f3483da6b" (UID: "871d04cb-19f3-4fc4-abff-ee5f3483da6b"). InnerVolumeSpecName "kube-api-access-f6qdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:47.005374 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.005353 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tls-certs" pod "871d04cb-19f3-4fc4-abff-ee5f3483da6b" (UID: "871d04cb-19f3-4fc4-abff-ee5f3483da6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:47.005469 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.005390 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-dshm" pod "871d04cb-19f3-4fc4-abff-ee5f3483da6b" (UID: "871d04cb-19f3-4fc4-abff-ee5f3483da6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.005526 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.005501 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af12066d-62f8-49e1-a886-7eb0b1bfae07-tls-certs" pod "af12066d-62f8-49e1-a886-7eb0b1bfae07" (UID: "af12066d-62f8-49e1-a886-7eb0b1bfae07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:47.005622 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.005603 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af12066d-62f8-49e1-a886-7eb0b1bfae07-kube-api-access-k94xh" pod "af12066d-62f8-49e1-a886-7eb0b1bfae07" (UID: "af12066d-62f8-49e1-a886-7eb0b1bfae07"). InnerVolumeSpecName "kube-api-access-k94xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:47.006508 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.006487 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-dshm" pod "af12066d-62f8-49e1-a886-7eb0b1bfae07" (UID: "af12066d-62f8-49e1-a886-7eb0b1bfae07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:47.104762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.104729 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k94xh\" (UniqueName: \"kubernetes.io/projected/af12066d-62f8-49e1-a886-7eb0b1bfae07-kube-api-access-k94xh\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.104762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.104755 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af12066d-62f8-49e1-a886-7eb0b1bfae07-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.104762 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.104766 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-home\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.104991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.104774 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af12066d-62f8-49e1-a886-7eb0b1bfae07-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.104991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.104782 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/871d04cb-19f3-4fc4-abff-ee5f3483da6b-tls-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.104991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.104789 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/871d04cb-19f3-4fc4-abff-ee5f3483da6b-dshm\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.104991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.104797 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6qdc\" (UniqueName: \"kubernetes.io/projected/871d04cb-19f3-4fc4-abff-ee5f3483da6b-kube-api-access-f6qdc\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.235706 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.235668 4433 generic.go:356] "Generic (PLEG): container finished" podID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerID="e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e" exitCode=0 Oct 30 00:40:47.235874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.235712 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" Oct 30 00:40:47.235874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.235755 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" event={"ID":"af12066d-62f8-49e1-a886-7eb0b1bfae07","Type":"ContainerDied","Data":"e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e"} Oct 30 00:40:47.235874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.235796 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd" event={"ID":"af12066d-62f8-49e1-a886-7eb0b1bfae07","Type":"ContainerDied","Data":"939533f767e8230859893a831efb0c6fd5518e865d24ffde78ec7785104397ca"} Oct 30 00:40:47.235874 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.235819 4433 scope.go:122] "RemoveContainer" containerID="e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e" Oct 30 00:40:47.237073 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.237055 4433 generic.go:356] "Generic (PLEG): container finished" podID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerID="615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946" exitCode=0 Oct 30 00:40:47.237184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.237085 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" event={"ID":"871d04cb-19f3-4fc4-abff-ee5f3483da6b","Type":"ContainerDied","Data":"615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946"} Oct 30 00:40:47.237184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.237108 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" Oct 30 00:40:47.237184 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.237120 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth" event={"ID":"871d04cb-19f3-4fc4-abff-ee5f3483da6b","Type":"ContainerDied","Data":"2f53dc4827f035345a4da56ea0df3aab92d7cd4e2aee3d4ef297d3add394f290"} Oct 30 00:40:47.244857 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.244836 4433 scope.go:122] "RemoveContainer" containerID="e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e" Oct 30 00:40:47.245127 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:40:47.245108 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e\": container with ID starting with e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e not found: ID does not exist" containerID="e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e" Oct 30 00:40:47.245173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.245134 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e"} err="failed to get container status \"e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e\": rpc error: code = NotFound desc = could not find container \"e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e\": container with ID starting with e9ed86af694cb7e69bc0a4166c61d8a78c266c646d23b185e1ff4130cc0a1e2e not found: ID does not exist" Oct 30 00:40:47.245173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.245153 4433 scope.go:122] "RemoveContainer" containerID="615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946" Oct 30 00:40:47.253052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.253034 4433 scope.go:122] "RemoveContainer" containerID="615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946" Oct 30 00:40:47.253312 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:40:47.253293 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946\": container with ID starting with 615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946 not found: ID does not exist" containerID="615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946" Oct 30 00:40:47.253364 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.253318 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946"} err="failed to get container status \"615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946\": rpc error: code = NotFound desc = could not find container \"615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946\": container with ID starting with 615fb483d9e18c865f2e120966a82e8d67694a3fc5efdb082060de2269b71946 not found: ID does not exist" Oct 30 00:40:47.289265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.289240 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth"] Oct 30 00:40:47.302428 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.302403 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/distinct-w6-6c45bb-kserve-648b9f4887-hcxth"] Oct 30 00:40:47.328986 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.328958 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd"] Oct 30 00:40:47.332687 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:47.332664 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/premium-w6-6c45bb-kserve-8d4fc8b48-frlmd"] Oct 30 00:40:49.177718 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:49.177684 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" path="/var/lib/kubelet/pods/871d04cb-19f3-4fc4-abff-ee5f3483da6b/volumes" Oct 30 00:40:49.178131 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:49.178115 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" path="/var/lib/kubelet/pods/af12066d-62f8-49e1-a886-7eb0b1bfae07/volumes" Oct 30 00:40:53.798605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:53.798522 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4"] Oct 30 00:40:53.799059 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:53.798738 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" podUID="71148984-2b26-4b03-86c5-8d1f56dfaf17" containerName="maas-api" containerID="cri-o://0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227" gracePeriod=30 Oct 30 00:40:53.869083 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:53.869049 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:54.071773 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.071744 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:40:54.179845 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.179812 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71148984-2b26-4b03-86c5-8d1f56dfaf17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71148984-2b26-4b03-86c5-8d1f56dfaf17-openshift-service-ca\") pod \"71148984-2b26-4b03-86c5-8d1f56dfaf17\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " Oct 30 00:40:54.180046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.179885 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71148984-2b26-4b03-86c5-8d1f56dfaf17-kube-api-access-lqxxx\" (UniqueName: \"kubernetes.io/projected/71148984-2b26-4b03-86c5-8d1f56dfaf17-kube-api-access-lqxxx\") pod \"71148984-2b26-4b03-86c5-8d1f56dfaf17\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " Oct 30 00:40:54.180046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.179932 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls\") pod \"71148984-2b26-4b03-86c5-8d1f56dfaf17\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " Oct 30 00:40:54.180046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.179993 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-metrics-certs\") pod \"71148984-2b26-4b03-86c5-8d1f56dfaf17\" (UID: \"71148984-2b26-4b03-86c5-8d1f56dfaf17\") " Oct 30 00:40:54.180281 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.180245 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71148984-2b26-4b03-86c5-8d1f56dfaf17-openshift-service-ca" pod "71148984-2b26-4b03-86c5-8d1f56dfaf17" (UID: "71148984-2b26-4b03-86c5-8d1f56dfaf17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:40:54.182042 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.182016 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls" pod "71148984-2b26-4b03-86c5-8d1f56dfaf17" (UID: "71148984-2b26-4b03-86c5-8d1f56dfaf17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:54.182138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.182035 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-metrics-certs" pod "71148984-2b26-4b03-86c5-8d1f56dfaf17" (UID: "71148984-2b26-4b03-86c5-8d1f56dfaf17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:54.182138 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.182042 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71148984-2b26-4b03-86c5-8d1f56dfaf17-kube-api-access-lqxxx" pod "71148984-2b26-4b03-86c5-8d1f56dfaf17" (UID: "71148984-2b26-4b03-86c5-8d1f56dfaf17"). InnerVolumeSpecName "kube-api-access-lqxxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:54.276177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.276141 4433 generic.go:356] "Generic (PLEG): container finished" podID="71148984-2b26-4b03-86c5-8d1f56dfaf17" containerID="0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227" exitCode=0 Oct 30 00:40:54.276177 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.276182 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" Oct 30 00:40:54.276484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.276227 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" event={"ID":"71148984-2b26-4b03-86c5-8d1f56dfaf17","Type":"ContainerDied","Data":"0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227"} Oct 30 00:40:54.276484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.276284 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" event={"ID":"71148984-2b26-4b03-86c5-8d1f56dfaf17","Type":"ContainerDied","Data":"64357cc9e4dce9a291b4d897a908665248adb34027af709918b7b6889ad62a03"} Oct 30 00:40:54.276484 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.276314 4433 scope.go:122] "RemoveContainer" containerID="0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227" Oct 30 00:40:54.281787 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.281763 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqxxx\" (UniqueName: \"kubernetes.io/projected/71148984-2b26-4b03-86c5-8d1f56dfaf17-kube-api-access-lqxxx\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:54.281919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.281789 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-maas-api-tls\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:54.281919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.281803 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71148984-2b26-4b03-86c5-8d1f56dfaf17-metrics-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:54.281919 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.281816 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71148984-2b26-4b03-86c5-8d1f56dfaf17-openshift-service-ca\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:40:54.291780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.291752 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:54.296323 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.296305 4433 scope.go:122] "RemoveContainer" containerID="0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227" Oct 30 00:40:54.296670 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:40:54.296650 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227\": container with ID starting with 0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227 not found: ID does not exist" containerID="0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227" Oct 30 00:40:54.296748 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.296678 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227"} err="failed to get container status \"0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227\": rpc error: code = NotFound desc = could not find container \"0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227\": container with ID starting with 0dfa579d77667dca1420f593991797e30e2221eb24ee1b85d70b82f99ba94227 not found: ID does not exist" Oct 30 00:40:54.330805 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.330779 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4"] Oct 30 00:40:54.336811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.336789 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4"] Oct 30 00:40:54.417413 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.417380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:54.999459 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:54.999422 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:55.178025 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:55.177994 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71148984-2b26-4b03-86c5-8d1f56dfaf17" path="/var/lib/kubelet/pods/71148984-2b26-4b03-86c5-8d1f56dfaf17/volumes" Oct 30 00:40:57.027568 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:40:57.027526 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6c45bb-5bdd4d7f76-849z4" podUID="71148984-2b26-4b03-86c5-8d1f56dfaf17" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.53:8443/health\": context deadline exceeded" Oct 30 00:41:00.734265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:00.734222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:00.840990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:00.840949 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:01.060896 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:01.060853 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx"] Oct 30 00:41:01.061171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:01.061143 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" containerID="cri-o://9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b" gracePeriod=30 Oct 30 00:41:01.095707 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:01.095671 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:04.784781 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:04.784751 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:41:05.886829 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:05.886792 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:41:06.053303 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.053275 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:41:06.321394 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.321370 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:41:06.336447 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.336420 4433 generic.go:356] "Generic (PLEG): container finished" podID="c51c8a69-4712-4df7-a958-7f53f9338811" containerID="9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b" exitCode=0 Oct 30 00:41:06.336606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.336465 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" Oct 30 00:41:06.336606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.336513 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" event={"ID":"c51c8a69-4712-4df7-a958-7f53f9338811","Type":"ContainerDied","Data":"9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b"} Oct 30 00:41:06.336606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.336581 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx" event={"ID":"c51c8a69-4712-4df7-a958-7f53f9338811","Type":"ContainerDied","Data":"ec239921e2fcd684d2b62e1e8ee63298aafef91eb7b6c54ec5f21d843c09f0a2"} Oct 30 00:41:06.336606 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.336604 4433 scope.go:122] "RemoveContainer" containerID="9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b" Oct 30 00:41:06.345561 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.345540 4433 scope.go:122] "RemoveContainer" containerID="9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b" Oct 30 00:41:06.345825 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:41:06.345804 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b\": container with ID starting with 9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b not found: ID does not exist" containerID="9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b" Oct 30 00:41:06.345894 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.345837 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b"} err="failed to get container status \"9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b\": rpc error: code = NotFound desc = could not find container \"9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b\": container with ID starting with 9e289f64b78a147bf31fc31472c0d3dde54f7a3f522b9318bcfc95fbd5b9a14b not found: ID does not exist" Oct 30 00:41:06.400983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.400955 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-kube-api-access-4npgf\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-kube-api-access-4npgf\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401042 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-envoy\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401077 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-istio-token\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-istio-token\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401103 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c51c8a69-4712-4df7-a958-7f53f9338811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c8a69-4712-4df7-a958-7f53f9338811-istio-podinfo\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401135 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401120 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-certs\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401184 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-socket\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401212 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51c8a69-4712-4df7-a958-7f53f9338811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c8a69-4712-4df7-a958-7f53f9338811-istiod-ca-cert\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401243 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-data\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401375 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401270 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-credential-socket\") pod \"c51c8a69-4712-4df7-a958-7f53f9338811\" (UID: \"c51c8a69-4712-4df7-a958-7f53f9338811\") " Oct 30 00:41:06.401618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401526 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-socket" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.401709 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401663 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.401776 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401734 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-credential-socket" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.401907 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401886 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-data" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.401991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401914 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-certs" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.402050 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.401998 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51c8a69-4712-4df7-a958-7f53f9338811-istiod-ca-cert" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:41:06.403806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.403769 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-kube-api-access-4npgf" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "kube-api-access-4npgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:06.403806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.403791 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c51c8a69-4712-4df7-a958-7f53f9338811-istio-podinfo" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:41:06.403806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.403804 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-istio-token" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:06.404046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.404031 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-envoy" pod "c51c8a69-4712-4df7-a958-7f53f9338811" (UID: "c51c8a69-4712-4df7-a958-7f53f9338811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.469306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.469215 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:41:06.503991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.503962 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-envoy\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.503991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.503991 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-istio-token\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.504170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.504000 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c8a69-4712-4df7-a958-7f53f9338811-istio-podinfo\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.504170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.504008 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-workload-certs\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.504170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.504016 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c8a69-4712-4df7-a958-7f53f9338811-istiod-ca-cert\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.504170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.504026 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-istio-data\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.504170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.504033 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c8a69-4712-4df7-a958-7f53f9338811-credential-socket\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.504170 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.504042 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npgf\" (UniqueName: \"kubernetes.io/projected/c51c8a69-4712-4df7-a958-7f53f9338811-kube-api-access-4npgf\") on node \"ip-10-0-138-81.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.685795 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.685747 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx"] Oct 30 00:41:06.687618 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:06.687587 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-p4fbx"] Oct 30 00:41:07.098181 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:41:07.098150 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:41:07.178173 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:07.178141 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" path="/var/lib/kubelet/pods/c51c8a69-4712-4df7-a958-7f53f9338811/volumes" Oct 30 00:41:08.456837 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:08.456782 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:08.565851 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:08.565814 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:09.221820 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:09.221780 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:14.924408 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:14.924375 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:18.271420 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:18.271389 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:41:19.371330 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:19.371294 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:41:19.556165 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:19.556135 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:41:19.964790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:19.964757 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:41:21.104171 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:21.104140 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66d7794744-s8p8s_6f13b10d-a3d2-4e2f-83ff-f74e51e7849b/maas-api/0.log" Oct 30 00:41:24.706052 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:24.706019 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-vwjgw_44e25d2a-4611-4532-b4db-c28d758ee4f1/postgres/0.log" Oct 30 00:41:29.172901 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:29.172862 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vtwgb_db7f5c4a-f002-4e25-8c88-51082bf3417d/discovery/0.log" Oct 30 00:41:29.538290 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:29.538206 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-th2sw_27fad148-1b28-4abc-9bb0-11a38d22f28e/kube-auth-proxy/0.log" Oct 30 00:41:29.716941 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:29.716911 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-cj97z_e0d5fcfc-fadc-44e6-9b61-986bc7395aca/istio-proxy/0.log" Oct 30 00:41:30.451036 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:30.451000 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fhm5v_90ed5b12-1185-4013-a1f1-4d98879d0efb/payload-pre-processing/0.log" Oct 30 00:41:30.626154 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:30.626120 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log" Oct 30 00:41:31.784223 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:31.784193 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-jvcs4_12e5f30f-344a-4a0c-82ac-60ceb5fbebf2/main/0.log" Oct 30 00:41:31.980005 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:31.979976 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bw4ln_f6aae325-01b5-4ab6-8360-c0e67efd8e43/main/0.log" Oct 30 00:41:32.169997 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:32.169965 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6z2n8_6b2916e1-b456-4941-a17e-235744eac859/main/0.log" Oct 30 00:41:32.549024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:32.548944 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qs69k_8e26ca3e-0c37-482b-81e5-ac51211ad7b6/main/0.log" Oct 30 00:41:32.911524 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:32.911488 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-spdzp_059340ab-5d93-4759-8d08-b5203c258e05/main/0.log" Oct 30 00:41:33.269341 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:33.269244 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dmg2v_de398e8d-5cd0-4e2f-ae60-7bf17a4a942e/main/0.log" Oct 30 00:41:37.097483 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:41:37.097406 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:41:40.688952 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:40.688916 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:41.266790 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:41.266757 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:41.378887 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:41.378852 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:43.602702 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.602655 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:43.679180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679138 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w9p6f/must-gather-zrxpz"] Oct 30 00:41:43.679678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679658 4433 cpu_manager.go:414] "containerMap: removing container" podUID="71148984-2b26-4b03-86c5-8d1f56dfaf17" containerName="maas-api" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679679 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71148984-2b26-4b03-86c5-8d1f56dfaf17" containerName="maas-api" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679699 4433 cpu_manager.go:414] "containerMap: removing container" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679706 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679718 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679725 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679735 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679743 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679756 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679763 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679774 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679781 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" Oct 30 00:41:43.679796 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679793 4433 cpu_manager.go:414] "containerMap: removing container" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679802 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679813 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679819 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679835 4433 cpu_manager.go:414] "containerMap: removing container" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679844 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679853 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679861 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679870 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679878 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679896 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679904 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679916 4433 cpu_manager.go:414] "containerMap: removing container" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679923 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679938 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679945 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679956 4433 cpu_manager.go:414] "containerMap: removing container" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.679996 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680010 4433 cpu_manager.go:414] "containerMap: removing container" podUID="99095068-2c79-4c7f-a79b-5c4e8a550872" containerName="test-curl-7593-521715" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680017 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99095068-2c79-4c7f-a79b-5c4e8a550872" containerName="test-curl-7593-521715" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680028 4433 cpu_manager.go:414] "containerMap: removing container" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680037 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680052 4433 cpu_manager.go:414] "containerMap: removing container" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680059 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680070 4433 cpu_manager.go:414] "containerMap: removing container" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680076 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680088 4433 cpu_manager.go:414] "containerMap: removing container" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680095 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680137 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2f92ed7a-b83b-47e0-8778-573e8f1d6efc" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680157 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f92ed7a-b83b-47e0-8778-573e8f1d6efc" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680174 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680182 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680193 4433 cpu_manager.go:414] "containerMap: removing container" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680200 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680209 4433 cpu_manager.go:414] "containerMap: removing container" podUID="58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680216 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680231 4433 cpu_manager.go:414] "containerMap: removing container" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680239 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680248 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3b5cc49e-a720-4888-ab38-37f7380c3d5f" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680254 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5cc49e-a720-4888-ab38-37f7380c3d5f" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680264 4433 cpu_manager.go:414] "containerMap: removing container" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680272 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680287 4433 cpu_manager.go:414] "containerMap: removing container" podUID="c9ad66e9-958f-4411-9c86-542cb7739541" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680295 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ad66e9-958f-4411-9c86-542cb7739541" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680312 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680319 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680330 4433 cpu_manager.go:414] "containerMap: removing container" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680336 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680350 4433 cpu_manager.go:414] "containerMap: removing container" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680357 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680369 4433 cpu_manager.go:414] "containerMap: removing container" podUID="e7f89b3b-8a1f-4b55-bff9-c22231711f78" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680376 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f89b3b-8a1f-4b55-bff9-c22231711f78" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680386 4433 cpu_manager.go:414] "containerMap: removing container" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680393 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680403 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680410 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680422 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680430 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680543 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680557 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f89b3b-8a1f-4b55-bff9-c22231711f78" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680566 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="af12066d-62f8-49e1-a886-7eb0b1bfae07" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680591 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4deb7d9-4a19-418f-a8c3-9d3064257bfd" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680599 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e586cc2-705e-44fb-abaa-dd8993da8faf" containerName="main" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680608 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="71148984-2b26-4b03-86c5-8d1f56dfaf17" containerName="maas-api" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680618 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="75459eb3-1ef0-4e11-88cc-bd9ce9c3ac2d" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680630 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d990bb4a-b08e-49de-ba43-2181a0996cea" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680638 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d12bf49-df66-40a5-b4a6-74163b6a88ed" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680648 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade4e4dc-fc17-4e27-901f-9321d6ea55e4" containerName="istio-proxy" Oct 30 00:41:43.680617 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680656 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f69cc2-29c2-47f4-8b74-34ace130d3d8" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680665 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="871d04cb-19f3-4fc4-abff-ee5f3483da6b" containerName="main" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680672 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffa82da-a0b6-473b-9556-a11cd7e0a0b9" containerName="main" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680681 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6ca814-0703-421b-afdd-dfe9b05555ea" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680689 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fb91fd-aa1a-43a5-a388-24ba8562fa23" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680696 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3993addf-2a45-44a7-9714-ad89b90ea081" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680704 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680713 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb6c7a9-f3dd-4f0d-8081-d274df193a9d" containerName="main" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680722 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5c6a4d-05ab-4ba0-aaa9-c6969a303499" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680731 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680740 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="99095068-2c79-4c7f-a79b-5c4e8a550872" containerName="test-curl-7593-521715" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680749 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="191e9810-53a6-41a4-b6fd-aa2f8b9a5faa" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680757 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="26be440e-117f-46bd-957c-f59d9616abcb" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680765 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f65f05-73d3-4d40-81f7-3300f86869ef" containerName="main" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680773 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c8a69-4712-4df7-a958-7f53f9338811" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680782 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f92ed7a-b83b-47e0-8778-573e8f1d6efc" containerName="main" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680791 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d44d38e-54b0-473f-8f81-64328091c5f7" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680800 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="643500a6-e1c6-4483-a4f6-ae816b5b8987" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680807 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bbbf0e-a33c-40b1-82bd-7a73bb9699ad" containerName="maas-api" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680816 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="735ced8b-6586-4e93-8d8a-3439fab15f89" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680824 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5cc49e-a720-4888-ab38-37f7380c3d5f" containerName="maas-api" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680831 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="34827380-65c5-479c-bf82-35dfbd7bc003" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680840 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ad66e9-958f-4411-9c86-542cb7739541" containerName="maas-api" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.680850 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f1b969-18ae-48bf-bc75-b859ab445850" containerName="istio-proxy" Oct 30 00:41:43.683438 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.681067 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="509d7476-b7d1-4657-b779-a2876047d78c" containerName="istio-proxy" Oct 30 00:41:43.684751 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.684732 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:43.700443 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.700419 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9p6f\"/\"openshift-service-ca.crt\"" Oct 30 00:41:43.703213 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.703192 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9p6f/must-gather-zrxpz"] Oct 30 00:41:43.711347 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.711322 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:43.714057 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.714036 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w9p6f\"/\"default-dockercfg-2hg8z\"" Oct 30 00:41:43.714149 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.714124 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9p6f\"/\"kube-root-ca.crt\"" Oct 30 00:41:43.774447 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.774416 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkch\" (UniqueName: \"kubernetes.io/projected/6d8df278-aca4-463c-bec9-a1be49b1021f-kube-api-access-sdkch\") pod \"must-gather-zrxpz\" (UID: \"6d8df278-aca4-463c-bec9-a1be49b1021f\") " pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:43.774558 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.774535 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6d8df278-aca4-463c-bec9-a1be49b1021f-must-gather-output\") pod \"must-gather-zrxpz\" (UID: \"6d8df278-aca4-463c-bec9-a1be49b1021f\") " pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:43.875312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.875249 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkch\" (UniqueName: \"kubernetes.io/projected/6d8df278-aca4-463c-bec9-a1be49b1021f-kube-api-access-sdkch\") pod \"must-gather-zrxpz\" (UID: \"6d8df278-aca4-463c-bec9-a1be49b1021f\") " pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:43.875312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.875310 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6d8df278-aca4-463c-bec9-a1be49b1021f-must-gather-output\") pod \"must-gather-zrxpz\" (UID: \"6d8df278-aca4-463c-bec9-a1be49b1021f\") " pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:43.875629 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.875612 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6d8df278-aca4-463c-bec9-a1be49b1021f-must-gather-output\") pod \"must-gather-zrxpz\" (UID: \"6d8df278-aca4-463c-bec9-a1be49b1021f\") " pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:43.899232 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.899208 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkch\" (UniqueName: \"kubernetes.io/projected/6d8df278-aca4-463c-bec9-a1be49b1021f-kube-api-access-sdkch\") pod \"must-gather-zrxpz\" (UID: \"6d8df278-aca4-463c-bec9-a1be49b1021f\") " pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:43.998536 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:43.998510 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9p6f/must-gather-zrxpz" Oct 30 00:41:44.298194 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:44.298171 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9p6f/must-gather-zrxpz"] Oct 30 00:41:44.300182 ip-10-0-138-81 kubenswrapper[4433]: W1030 00:41:44.300156 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8df278_aca4_463c_bec9_a1be49b1021f.slice/crio-8ecf2215eb13776e30fdacbf92b69347534c0c907c29281bcd7bbfb77d0ea52b WatchSource:0}: Error finding container 8ecf2215eb13776e30fdacbf92b69347534c0c907c29281bcd7bbfb77d0ea52b: Status 404 returned error can't find the container with id 8ecf2215eb13776e30fdacbf92b69347534c0c907c29281bcd7bbfb77d0ea52b Oct 30 00:41:44.302180 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:44.302159 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:41:44.526716 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:44.526682 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9p6f/must-gather-zrxpz" event={"ID":"6d8df278-aca4-463c-bec9-a1be49b1021f","Type":"ContainerStarted","Data":"8ecf2215eb13776e30fdacbf92b69347534c0c907c29281bcd7bbfb77d0ea52b"} Oct 30 00:41:48.794940 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:48.794890 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2c7g8_97b31541-b41e-482e-bda2-afb18784d5d4/global-pull-secret-syncer/0.log" Oct 30 00:41:49.565179 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:49.565129 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9p6f/must-gather-zrxpz" event={"ID":"6d8df278-aca4-463c-bec9-a1be49b1021f","Type":"ContainerStarted","Data":"5f2095562f0497e073c675bfd9cc5a7fae90364d52c53d58786521986db5f0e9"} Oct 30 00:41:49.712292 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:49.712209 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" Oct 30 00:41:50.229995 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:50.229959 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-81.ec2.internal_0e8ee27d044b1e7c2240420e5926cb72/haproxy/0.log" Oct 30 00:41:50.572439 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:50.572400 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9p6f/must-gather-zrxpz" event={"ID":"6d8df278-aca4-463c-bec9-a1be49b1021f","Type":"ContainerStarted","Data":"cd1736f01c71f4131d7096a0b0209cb2a4a2b74a9f09b9d385bb8c3809ea2974"} Oct 30 00:41:50.613252 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:50.613202 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w9p6f/must-gather-zrxpz" podStartSLOduration=2.500823076 podStartE2EDuration="7.613190695s" totalImagesPullingTime="5.112367619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:50.613089944 +0000 UTC m=+2219.547158478" watchObservedRunningTime="2026-10-30 00:41:50.613190695 +0000 UTC m=+2219.547259229" Oct 30 00:41:51.259306 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:51.259273 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:41:53.006855 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:53.006802 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:41:53.040422 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:53.040388 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:41:54.011471 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:54.011423 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:41:55.268552 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:55.268520 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:41:56.569506 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:56.569473 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:41:56.610911 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:56.610879 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:41:57.836207 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:57.836176 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:41:58.505117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:58.505087 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:41:59.058678 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:59.058644 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:41:59.568806 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:59.568772 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:41:59.602725 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:59.602694 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:41:59.761705 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:59.761669 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df2499fe-2768-4988-9f24-4815a7d93fbc/alertmanager/0.log" Oct 30 00:41:59.880024 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:59.879919 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df2499fe-2768-4988-9f24-4815a7d93fbc/config-reloader/0.log" Oct 30 00:41:59.998437 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:41:59.998405 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df2499fe-2768-4988-9f24-4815a7d93fbc/kube-rbac-proxy-web/0.log" Oct 30 00:42:00.121890 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.121851 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4"] Oct 30 00:42:00.126132 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.126101 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df2499fe-2768-4988-9f24-4815a7d93fbc/kube-rbac-proxy/0.log" Oct 30 00:42:00.129007 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.128986 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.149430 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.149399 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4"] Oct 30 00:42:00.154886 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.154852 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:42:00.196611 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.196550 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:42:00.238826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.238772 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pr5qw\"/\"openshift-service-ca.crt\"" Oct 30 00:42:00.238826 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.238797 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pr5qw\"/\"kube-root-ca.crt\"" Oct 30 00:42:00.239202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.238899 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pr5qw\"/\"default-dockercfg-ms8r5\"" Oct 30 00:42:00.243786 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.243762 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df2499fe-2768-4988-9f24-4815a7d93fbc/kube-rbac-proxy-metric/0.log" Oct 30 00:42:00.247030 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.247003 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-podres\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.247146 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.247039 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-lib-modules\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.247261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.247155 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6s4r\" (UniqueName: \"kubernetes.io/projected/78bae9f7-9b3d-4e60-8258-799e26c1bef8-kube-api-access-m6s4r\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.247261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.247234 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-proc\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.247261 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.247257 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-sys\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348422 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-podres\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348463 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-lib-modules\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348498 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6s4r\" (UniqueName: \"kubernetes.io/projected/78bae9f7-9b3d-4e60-8258-799e26c1bef8-kube-api-access-m6s4r\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348603 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-podres\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348618 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-lib-modules\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348609 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-proc\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348652 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-proc\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348750 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348671 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-sys\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.348990 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.348771 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78bae9f7-9b3d-4e60-8258-799e26c1bef8-sys\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.361988 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.361959 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df2499fe-2768-4988-9f24-4815a7d93fbc/prom-label-proxy/0.log" Oct 30 00:42:00.373419 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.373391 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6s4r\" (UniqueName: \"kubernetes.io/projected/78bae9f7-9b3d-4e60-8258-799e26c1bef8-kube-api-access-m6s4r\") pod \"perf-node-gather-daemonset-frxc4\" (UID: \"78bae9f7-9b3d-4e60-8258-799e26c1bef8\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.447039 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.446971 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:00.480972 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.480948 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df2499fe-2768-4988-9f24-4815a7d93fbc/init-config-reloader/0.log" Oct 30 00:42:00.763495 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.763463 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4"] Oct 30 00:42:00.824650 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:00.824628 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:42:01.103955 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.103920 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b574c8449-xhpkt_de75ec79-9b11-40ce-8523-98f3e69e814a/metrics-server/0.log" Oct 30 00:42:01.224733 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.224702 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-bfvl5_f20dc4d0-e7fe-4204-9d6d-1b9ba2e20b8d/monitoring-plugin/0.log" Oct 30 00:42:01.402852 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.402778 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:42:01.643661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.643621 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" event={"ID":"78bae9f7-9b3d-4e60-8258-799e26c1bef8","Type":"ContainerStarted","Data":"41d5fb5c7ae425c42151f9bdba56b292a17c84e7b6acefb2032aed35930f193d"} Oct 30 00:42:01.643871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.643682 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" event={"ID":"78bae9f7-9b3d-4e60-8258-799e26c1bef8","Type":"ContainerStarted","Data":"fd943a0cb745d6755940960b3014622899de2527cfdcc854521925dd9eb20fdf"} Oct 30 00:42:01.643871 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.643738 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:01.685661 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.685549 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" podStartSLOduration=1.685535123 podStartE2EDuration="1.685535123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:01.684490409 +0000 UTC m=+2230.618558943" watchObservedRunningTime="2026-10-30 00:42:01.685535123 +0000 UTC m=+2230.619603656" Oct 30 00:42:01.716460 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.716429 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rc8jd_3d5c2cbc-2adb-4ad2-bb48-87071c4497db/node-exporter/0.log" Oct 30 00:42:01.829797 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.829765 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rc8jd_3d5c2cbc-2adb-4ad2-bb48-87071c4497db/kube-rbac-proxy/0.log" Oct 30 00:42:01.951888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:01.950498 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rc8jd_3d5c2cbc-2adb-4ad2-bb48-87071c4497db/init-textfile/0.log" Oct 30 00:42:02.213942 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:02.213846 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:42:02.422054 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:02.422023 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-vt9fp_f3e8bfb3-fa1f-4af9-81ec-32c946bc7919/kube-rbac-proxy-main/0.log" Oct 30 00:42:02.538543 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:02.538458 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-vt9fp_f3e8bfb3-fa1f-4af9-81ec-32c946bc7919/kube-rbac-proxy-self/0.log" Oct 30 00:42:02.654780 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:02.654747 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-vt9fp_f3e8bfb3-fa1f-4af9-81ec-32c946bc7919/openshift-state-metrics/0.log" Oct 30 00:42:03.216605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:03.216546 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:42:03.259900 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:03.259870 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:42:03.344350 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:03.344316 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:03.458844 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:03.458792 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghj4h_3740a0a7-2025-4cfb-aa4f-9bcbfaf68780/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:03.615605 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:03.615562 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4r6fr_cab57612-8d11-4773-a8c6-18c4fa22458a/prometheus-operator/0.log" Oct 30 00:42:03.839155 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:03.839122 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4r6fr_cab57612-8d11-4773-a8c6-18c4fa22458a/kube-rbac-proxy/0.log" Oct 30 00:42:03.958333 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:03.958257 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-mv64m_2e8b990f-be4c-434b-8966-5ebdb335e811/prometheus-operator-admission-webhook/0.log" Oct 30 00:42:04.082474 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.082437 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c77b8459-pk9h2_b28d23f1-0da1-4611-b347-0ff9fc50e4af/telemeter-client/0.log" Oct 30 00:42:04.199811 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.199739 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c77b8459-pk9h2_b28d23f1-0da1-4611-b347-0ff9fc50e4af/reload/0.log" Oct 30 00:42:04.320312 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.320267 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c77b8459-pk9h2_b28d23f1-0da1-4611-b347-0ff9fc50e4af/kube-rbac-proxy/0.log" Oct 30 00:42:04.447969 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.447937 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c75469d9-wzp7z_f1e6fb59-2134-4a4c-acd3-fe1fa27e6851/thanos-query/0.log" Oct 30 00:42:04.494556 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.494522 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:42:04.564265 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.564233 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c75469d9-wzp7z_f1e6fb59-2134-4a4c-acd3-fe1fa27e6851/kube-rbac-proxy-web/0.log" Oct 30 00:42:04.680867 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.680794 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c75469d9-wzp7z_f1e6fb59-2134-4a4c-acd3-fe1fa27e6851/kube-rbac-proxy/0.log" Oct 30 00:42:04.795405 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.795375 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c75469d9-wzp7z_f1e6fb59-2134-4a4c-acd3-fe1fa27e6851/prom-label-proxy/0.log" Oct 30 00:42:04.913402 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:04.913373 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c75469d9-wzp7z_f1e6fb59-2134-4a4c-acd3-fe1fa27e6851/kube-rbac-proxy-rules/0.log" Oct 30 00:42:05.028983 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:05.028898 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c75469d9-wzp7z_f1e6fb59-2134-4a4c-acd3-fe1fa27e6851/kube-rbac-proxy-metrics/0.log" Oct 30 00:42:05.997202 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:05.997170 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-697599cf87-kg92r_4ffe9b07-9588-402f-9578-039cdf337420/manager/0.log" Oct 30 00:42:06.987550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:06.987503 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vtwgb_db7f5c4a-f002-4e25-8c88-51082bf3417d/discovery/0.log" Oct 30 00:42:07.061011 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:07.060976 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f7bd4b46-9mpx5_04ba5f5f-a605-4090-82c2-6265d249c20b/manager/0.log" Oct 30 00:42:07.073977 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:07.073946 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-th2sw_27fad148-1b28-4abc-9bb0-11a38d22f28e/kube-auth-proxy/0.log" Oct 30 00:42:07.097720 ip-10-0-138-81 kubenswrapper[4433]: E1030 00:42:07.097691 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2mrhd" containerName="extractor" Oct 30 00:42:07.100025 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:07.100006 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w2kzg_79ecbf8c-f8a8-4cfa-884c-d066b189b266/server/0.log" Oct 30 00:42:07.129912 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:07.129890 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-cj97z_e0d5fcfc-fadc-44e6-9b61-986bc7395aca/istio-proxy/0.log" Oct 30 00:42:07.366176 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:07.366148 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fhm5v_90ed5b12-1185-4013-a1f1-4d98879d0efb/payload-pre-processing/0.log" Oct 30 00:42:07.419362 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:07.419330 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log" Oct 30 00:42:07.661636 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:07.661537 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-frxc4" Oct 30 00:42:08.101355 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:08.101320 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-h5kbp_9c408603-395a-45b8-8154-da48848349dc/proxy/0.log" Oct 30 00:42:08.843417 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:08.843383 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-567497877-228wm_43daf76b-958f-42f5-96f4-35007fe9ae74/console/0.log" Oct 30 00:42:08.977309 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:08.977273 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-hjwsx_0ef650f7-6cd3-4df0-8be4-5bc699dac77e/download-server/0.log" Oct 30 00:42:13.232117 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:13.232084 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2bm4_8f828c6e-3ce7-4457-b47a-fd34e484f0cd/dns/0.log" Oct 30 00:42:13.351991 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:13.351954 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2bm4_8f828c6e-3ce7-4457-b47a-fd34e484f0cd/kube-rbac-proxy/0.log" Oct 30 00:42:13.713703 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:13.713676 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8vxtm_8aad4aae-3822-4a95-aa08-ecc8fcce7bee/dns-node-resolver/0.log" Oct 30 00:42:14.156546 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:14.156510 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vtwgb_db7f5c4a-f002-4e25-8c88-51082bf3417d/discovery/0.log" Oct 30 00:42:14.249473 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:14.249445 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-th2sw_27fad148-1b28-4abc-9bb0-11a38d22f28e/kube-auth-proxy/0.log" Oct 30 00:42:14.309550 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:14.309513 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-cj97z_e0d5fcfc-fadc-44e6-9b61-986bc7395aca/istio-proxy/0.log" Oct 30 00:42:14.553017 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:14.552925 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fhm5v_90ed5b12-1185-4013-a1f1-4d98879d0efb/payload-pre-processing/0.log" Oct 30 00:42:14.610023 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:14.609995 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log" Oct 30 00:42:15.589046 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:15.589013 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q84nm_557dfb83-6842-4e18-b8e2-3426d7ba17ed/node-ca/0.log" Oct 30 00:42:18.518863 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:18.518829 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vtwgb_db7f5c4a-f002-4e25-8c88-51082bf3417d/discovery/0.log" Oct 30 00:42:18.612095 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:18.612061 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d88dd7bf-th2sw_27fad148-1b28-4abc-9bb0-11a38d22f28e/kube-auth-proxy/0.log" Oct 30 00:42:18.683388 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:18.683357 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-cj97z_e0d5fcfc-fadc-44e6-9b61-986bc7395aca/istio-proxy/0.log" Oct 30 00:42:18.934888 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:18.934852 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fhm5v_90ed5b12-1185-4013-a1f1-4d98879d0efb/payload-pre-processing/0.log" Oct 30 00:42:18.998370 ip-10-0-138-81 kubenswrapper[4433]: I1030 00:42:18.998340 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-sth25_4ba6aa75-07db-49e6-b604-d5c0ebd6133c/payload-processing/0.log"